CN111159129A

CN111159129A - Statistical method and device for log report

Info

Publication number: CN111159129A
Application number: CN201911423087.2A
Authority: CN
Inventors: 董泽奎; 郑彬; 陈亮; 梁大祥; 古来
Original assignee: NSFOCUS Information Technology Co Ltd; Nsfocus Technologies Inc
Current assignee: NSFOCUS Information Technology Co Ltd; Nsfocus Technologies Inc
Priority date: 2019-12-31
Filing date: 2019-12-31
Publication date: 2020-05-15

Abstract

The invention discloses a statistical method and a device of a log report, in the method, firstly, a field with lower probability in statistical service as a statistical object is used as a merging point, a plurality of logs are merged, and then the field corresponding to the statistical object of a first statistical service is inquired on the merged log according to the requirement of the first statistical service, thereby obtaining the log report corresponding to the first statistical service. After merging the logs, the number of the logs can be reduced, so that when the log report corresponding to the statistical service is obtained, the number of the logs to be inquired can be reduced, and the statistical mode can be ensured to be small in time delay.

Description

Statistical method and device for log report

Technical Field

The invention relates to the technical field of data processing, in particular to a statistical method and device of a log report.

Background

The enterprise security center is an equipment management platform, and is used for storing logs of each piece of equipment in an enterprise, such as firewall logs, intrusion fireproof logs or flow logs, and performing unified management and monitoring analysis on each piece of equipment in the enterprise based on the stored logs so as to ensure information security of the enterprise.

And monitoring and analyzing the equipment based on the stored logs, wherein the monitoring and analyzing process comprises a process of performing report statistics on the logs according to needs. For example, the flow log of a certain device may be counted to obtain a total flow report of the device in a preset time period; or, the firewall logs of a certain device may be counted, and a report of the number of times the device is intercepted in a preset time period may be obtained.

Because the number of devices in an enterprise is large, and the longer the service time of the devices is, the more and more logs are generated, the longer the time delay required for performing report statistics in a large number of logs is. To reduce the delay, one approach is to: the log of each device in the enterprise is stored in the distributed storage system, so that the distributed computing capability of the distributed storage system can be utilized to realize quick query, and the required log report can be obtained.

However, although the above method can obtain the log report faster, the method needs to configure the distributed storage system, and the consumption of hardware resources is large, so that the cost of counting the logs in this way is high. Therefore, a statistical method of log reports with low cost and small time delay is needed.

Disclosure of Invention

The invention provides a statistical method and a statistical device for a log report, which are used for providing a statistical mode for the log report with low cost and small time delay and solving the problem of high cost caused by the statistical mode for the log report in the prior art.

The invention provides a statistical method of log reports in a first aspect, which comprises the following steps:

determining a first field for merging the stored logs, wherein the first field is a field with a probability of being a statistical object of statistical traffic lower than a threshold;

merging the logs by taking the first field as a merging point and taking the residual fields as merging conditions to obtain a log group, wherein the log group comprises at least one merged log, and the residual fields are fields except the first field in the fields included in the logs;

acquiring a first statistical service, wherein the first statistical service comprises a first statistical object for performing report statistics on the plurality of logs;

and inquiring the value of a field corresponding to the first statistical object in each log of the log group to obtain a log report corresponding to the first statistical service.

In a possible implementation manner, the merging the multiple logs to obtain a log group, where the multiple logs are all traffic logs, the first field is a traffic field, the first field is a merging point, and remaining fields are merging conditions, and the merging includes:

determining a first log and a second log from the plurality of logs, wherein values of other fields except the flow field in the remaining fields respectively included in the first log and the second log are the same;

summing values of flow fields respectively included in the first log and the second log to obtain summed logs;

and obtaining the log group according to the log after summation processing and the log which is not subjected to the summation processing in the plurality of logs.

In a possible implementation manner, the merging the multiple logs to obtain a log group, where the multiple logs are all traffic logs, the first field is a traffic field and a time field, the first field is a merging point, and the remaining fields are merging conditions, and the merging includes:

summing values of flow fields respectively included by the first log and the second log, and combining time fields respectively included by the first log and the second log to obtain a summed log;

In a possible implementation manner, the merging the multiple logs by using the first field as a time field, the first field as a merging point, and the remaining fields as a merging condition to obtain a log group includes:

determining a first log and a second log from the plurality of logs, wherein values of other fields except the time field in the remaining fields included in the first log and the second log are the same;

merging the first log and the second log to obtain a third log, wherein the third log comprises all fields in the first log or the second log and also comprises a frequency field, and the value of the frequency field is the sum of the number of the first log and the second log;

and obtaining the log group according to the third log and the log which is not subjected to the merging processing in the plurality of logs.

In a possible implementation, the larger the sum of the numbers of the first and second logs, the smaller the number of logs included in the log group.

In a possible implementation, merging the plurality of logs to obtain a log group includes:

and merging the plurality of logs according to a preset period, wherein the log group obtained in the next period is obtained by merging the log group obtained in the previous period and at least one log newly added in the period.

The second aspect of the present invention provides a statistical apparatus for log reports, the apparatus comprising:

the processing unit is used for determining a first field for merging the stored logs, wherein the first field is a field with the probability of being taken as a statistical object of a statistical service and being lower than a threshold value; merging the logs by using the first field as a merging point and using a residual field as a merging condition to obtain a log group, wherein the log group comprises at least one merged log, and the residual field is a field except the first field in the fields included in the logs;

the acquiring unit is used for acquiring a first statistical service, wherein the first statistical service comprises a first statistical object for performing report statistics on the plurality of logs;

and the query unit is used for querying the value of the field corresponding to the first statistical object in each log of the log group to obtain a log report corresponding to the first statistical service.

In a possible implementation manner, the multiple logs are all traffic logs, the first field is a traffic field, the processing unit merges the multiple logs by using the first field as a merging point and using the remaining fields as a merging condition, so as to obtain a log group, where:

In a possible implementation manner, the multiple logs are all traffic logs, the first field is a traffic field and a time field, the processing unit merges the multiple logs by using the first field as a merging point and using the remaining fields as a merging condition, so as to obtain a log group, and is specifically configured to:

In a possible implementation manner, the multiple logs are all monitoring event logs, the first field is a time field, the processing unit merges the multiple logs by using the first field as a merging point and using the remaining fields as a merging condition, so as to obtain a log group, and is specifically configured to:

In a possible implementation manner, the processing unit merges the plurality of logs to obtain a log group, and is specifically configured to:

The third aspect of the present invention provides a statistical apparatus for log reports, the apparatus comprising:

at least one processor, and,

a memory communicatively coupled to the at least one processor, a communication interface;

wherein the memory stores instructions executable by the at least one processor, the at least one processor performing the method of any one of the first aspect using the communication interface by executing the instructions stored by the memory.

A fourth aspect of the invention provides a computer readable storage medium having stored thereon computer instructions which, when run on a computer, cause the computer to perform the method of any of the first aspects.

In a fifth aspect, an embodiment of the present invention further provides a computer program product, which, when run on an electronic device, causes the electronic device to execute a method that implements any one of the above first aspect and the first aspect of the embodiment of the present invention.

The technical scheme provided by the embodiment of the invention at least has the following beneficial effects:

when the report is unified according to the stored logs, firstly, a field with lower probability in the statistical service, which is taken as a statistical object, is taken as a merging point, the logs are merged, and then the field corresponding to the statistical object of the first statistical service is inquired on the merged log according to the requirement of the first statistical service, so that a log report corresponding to the first statistical service is obtained. After the logs are merged, the number of the logs can be reduced, so that the number of the logs to be inquired can be reduced when the log report corresponding to the statistical service is obtained, and the smaller time delay of the statistical mode can be ensured. And because the number of the logs needing to be inquired can be reduced, under the condition of the log quantity of the same scale, the required hardware resources can be reduced, the cost can be reduced, and the statistical mode of the log report with low cost and smaller time delay is realized.

Drawings

Fig. 1 is a flowchart illustrating an example of a statistical method for a log report according to an embodiment of the present invention;

fig. 2 is a schematic structural diagram of an example of a statistical apparatus for log reports provided in an embodiment of the present invention;

fig. 3 is a schematic structural diagram of another example of the statistical apparatus of the log report according to the embodiment of the present invention.

Detailed Description

The technical solutions of the present invention are described in detail below with reference to the drawings and the specific embodiments, and it should be understood that the specific features in the embodiments and the embodiments of the present invention are not intended to limit the technical solutions of the present invention, but may be combined with each other without conflict.

In order to better understand the technical scheme, the design idea of the embodiment of the invention is briefly introduced.

The number of logs stored in an enterprise security center can reach hundreds of millions, billions, or even billions of levels. How to rapidly perform report statistics on the logs with the magnitude is a problem to be solved by the enterprise security center.

In the related art, one solution is: the method adopts a set of system of big data and distributed computation, firstly, original logs are directly stored in a distributed database, when the logs need to be queried to obtain log reports, the distributed computation capability of the distributed database is utilized to realize rapid query, and therefore the log reports are obtained according to query results. Theoretically, as long as enough hardware resources are available, the larger log amount can be rapidly counted. However, the scheme needs to configure the distributed storage system, and has high consumption of hardware resources and high cost.

In order to increase the speed of acquiring the log report under the condition of limited hardware resources, an optimization direction may be how to pre-process a large amount of logs. In view of this, the present invention provides a statistical method for log reports, before performing report statistics, a field with a low probability of being a statistical object in a statistical service is used as a merging point to merge the multiple logs, and after the log is merged, the number of logs can be reduced, so that when obtaining a log report corresponding to the statistical service, the number of logs to be queried can be reduced, thereby ensuring that the time delay of the statistical method is small, and since the number of logs to be queried can be reduced, under the condition of the log quantity of the same scale, required hardware resources can be reduced, and the cost can be reduced.

For the above scenario, the following describes in detail the statistical method of the log report provided by the present invention with reference to the drawings of the specification. It should be noted that the method may be applied to an apparatus for managing an Enterprise device, for example, the apparatus may be an Enterprise Security center (ESPC) management platform, or may also be an apparatus for storing a log of an Enterprise device, which is not limited herein. Hereinafter, the method will be described by taking an example of its application to ESPC.

Referring to fig. 1, a flowchart of a statistical method for a log report according to an embodiment of the present invention is described as follows:

step 101, determining a first field for merging the stored logs, wherein the first field is a field with a probability of being a statistical object of the statistical service being lower than a threshold value.

In the ESPC, a log of all devices in the enterprise is stored, and the log may be a flow log recording a flow used by the device, or may be a firewall log recording an operation of the device intercepted by a firewall of the enterprise, or may be an access log recording a number of times that the device accesses a certain website, and the like, which is not limited herein. The ESPC can store the logs in a classified manner according to the contents of the logs, for example, all the flow logs in the enterprise are stored in one data table, and all the firewall logs in the enterprise are stored in another data table, so that in the subsequent use process, different data tables can be queried according to the requirements of statistical services, so that corresponding log reports can be obtained, and the processing speed is increased.

In the embodiment of the present invention, the plurality of logs in step 101 may be logs included in one of the plurality of data tables, for example, the plurality of logs may be a plurality of traffic logs included in a data table for storing traffic logs. Each traffic log may include a plurality of fields, for example, referring to table 1, a plurality of fields of a destination Internet Protocol (IP) address, a source IP address, a protocol type, a destination port, a source port, and a traffic value.

TABLE 1

Destination IP	Source IP	Type of protocol	Destination port	Source port	Flow rate value
						1.1.1.1	2.2.2.1	Tcp	443	43201	200

The ESPC then determines the first field to merge from the plurality of fields included in the traffic log. The first field may be a field, of the fields included in each log, whose probability of being a statistical object is lower than a threshold, where the threshold may be a smaller value, and a specific value is not limited. In the following example, the plurality of logs are taken as traffic logs as an example.

As an example, the first field may be preset, for example, a technician may determine, according to the statistical service corresponding to the ESPC, that the access time field in the plurality of fields is not to be a statistical object or has a low probability of being a statistical object, so as to set the first field as the access time field.

Here, the statistical object will be explained. The statistical object corresponds to the statistical service, each statistical service includes the statistical object, for example, a certain statistical service is to count the total traffic of each IP in one hour, and in the statistical service, the statistical object is the destination IP field.

As another example, the first field may be obtained by the ESPC by analyzing a statistical object corresponding to historical statistical traffic. For example, the ESPC obtains a plurality of historical statistical services within a preset time period, for example, the preset time period includes 3 statistical services, respectively, to count the total traffic of each destination IP within 30 minutes, count the total traffic of each destination IP and destination port within 5 minutes, and count the total traffic of each source IP and source port within 5 minutes. Since each of the plurality of statistical services is a total statistical traffic, that is, it is not concerned about which protocol type a certain traffic is, the ESPC determines that, of the 3 statistical services, only the protocol type is not taken as a statistical object, and thus determines that the first field is a protocol type field. In this case, the first field may be changed according to a plurality of statistical services performed by the ESPC, which increases flexibility of the scheme and may make the merging process more suitable for use.

In addition, the first field may be a field not included in the log. As an example, the first field may be a number field, for example, for a firewall log, each firewall log may be regarded as one interception of a firewall against a certain IP, and multiple firewall logs for the same IP may be regarded as multiple interception of the firewall against the IP, in this case, although the firewall log does not include the number field, a value of the number field may be obtained according to the number of the firewall logs, so that the number may be used as the first field for merging the firewall logs.

It should be noted that the number of the first field may be one, for example, for the aforementioned traffic log, the protocol type may be used as the first field; alternatively, the number of the first fields may be multiple, for example, for the aforementioned traffic log, both the protocol type and the time may be used as the first fields, and the number of the first fields is not limited herein.

In the embodiment of the present invention, if the ESPC includes multiple data tables, the first field for each data table may be determined, or the multiple logs in step 101 may also be all logs stored in the ESPC, and the ESPC performs unified management on all logs, and in these two cases, the determination manner of the first field is similar to the foregoing process, and details are not described here again.

And 102, merging the plurality of logs by taking the first field as a merging point and the rest fields as merging conditions to obtain a log group.

In this embodiment, the log group includes at least one merged log, and the remaining fields include fields other than the first field in a plurality of fields included in the plurality of logs. It should be noted that, as can be seen from the foregoing, the first field may be a field that is not included in the multiple logs, and in this case, the remaining fields are all fields included in the multiple logs.

And when the ESPC determines the first field, the ESPC takes the first field as a merging point to merge the plurality of logs. The process of this merging process will be described below with respect to two different types of logs. The two types of logs may be a flow log, or may be a monitoring event log, where the monitoring event may be a firewall log that records that a firewall of an enterprise intercepts operations of a device, or may be an access log that records the number of times that the device accesses a certain website, and the like, which are not examples herein.

For the traffic log:

as an example, the first field is a traffic field. When merging the logs, firstly determining a first log and a second log from the logs, wherein values of other fields except the flow field in the remaining fields respectively included in the first log and the second log are the same, then summing values of the flow fields respectively included in the first log and the second log to obtain a summed log, and finally obtaining the log group according to the summed log and the log which is not summed in the logs.

For example, the plurality of traffic logs are as shown in table 2, and 4 traffic logs are included in table 2. The first field is used as the traffic field, that is, the traffic field is a field with a low probability of being a statistical object of the statistical service. Then, using other fields in the flow log as merging conditions, and comparing whether the values of the other fields in the multiple flow logs are the same, for example, in table 2, the values of each of the other fields of the first flow log and the third flow log are the same except for the flow value, and then determining that the first flow log and the third flow log are the first log and the second log, so as to sum the flow values respectively included in the first log and the second log, for example, the flow value in the first log is 200, the flow value in the second log is 300, and the sum is 500, thereby obtaining a new flow log, as shown in table 3, the flow value in the new flow log is 500, and the values of the other fields are the same as the values of the first log or the second log.

TABLE 2

Destination IP	Source IP	Type of protocol	Destination port	Source port	Flow rate value
						1.1.1.1	2.2.2.1	Tcp	443	43201	200
1.1.1.2	2.2.2.2	Udp	53	32421	200
						1.1.1.1	2.2.2.1	Tcp	443	43201	300
1.1.1.2	3.3.3.4	Tcp	443	43242	300

TABLE 3

Then, a log group after merging processing is obtained according to the flow log not subjected to summing processing in table 2 and the new flow log in table 3, as shown in table 4. The flow logs in table 2 that are not summed are the second flow log and the fourth flow log, and table 3 includes a new flow log, and these 3 flow logs form the log group shown in table 4.

TABLE 4

Destination IP	Source IP	Type of protocol	Destination port	Source port	Flow rate value
						1.1.1.1	2.2.2.1	Tcp	443	43201	500
1.1.1.2	2.2.2.2	Udp	53	32421	200
						1.1.1.2	3.3.3.4	Tcp	443	43242	300

As can be seen from tables 2 and 4, table 2 includes 4 flow logs before merging; after merging, only 3 traffic logs are included in table 4, and thus the number of traffic logs is reduced after merging, so that the number of traffic logs to be queried can be reduced when query is performed according to statistical services subsequently, and processing delay can be reduced.

As another example, if the time field is included in the log, and the time field in each log is always ignored when merging is performed, the first field may be the traffic field and the time field. For example, the plurality of traffic logs are shown in table 5, and different from the traffic log shown in table 2, the traffic log shown in table 5 further includes a time field.

TABLE 5

Destination IP	Source IP	Type of protocol	Destination port	Source port	Time of day	Flow rate value
							1.1.1.1	2.2.2.1	Tcp	443	43201	2019-10-18 10:05	200
1.1.1.2	2.2.2.2	Udp	53	32421	2019-10-18 10:10	200
							1.1.1.1	2.2.2.1	Tcp	443	43201	2019-10-18 10:15	300
1.1.1.2	3.3.3.4	Tcp	443	43242	2019-10-18 10:25	300

In table 5, the values of the first flow log and the third flow log are the same except that the values of the flow field and the time field are different, so that the flow values included in the first flow log and the third flow log are summed, and the values of the time field are combined to obtain a new flow log. Wherein the procedure for the flow values is the same as in the previous example, resulting in a flow value sum of 500. The time field is merged, for example, the value of the time field in the first traffic log is 2019-10-1810:05, the value of the time field in the third traffic log is 2019-10-1810:15, and the maximum value is 2019-10-1810:15, so that the value of the merged time field is 2019-10-1810:15, and of course, other merging manners may be adopted for the time field, which is not limited herein. As shown in table 6, the new flow log has a flow value of 500, a time field of 2019-10-1810:15, and other fields identical to the first log or the second log.

TABLE 6

Destination IP

Source IP

Type of protocol

Destination port

Source port

Time of day

Flow rate value

1.1.1.1

2.2.2.1

Tcp

443

43201

2019-10-18 10:15

500

For monitoring event logs:

as an example, the first field is a time field, when merging the multiple logs, first, a first log and a second log are determined from the multiple logs, values of other fields except for the time field in the remaining fields included in the first log and the second log respectively are the same, then, the first log and the second log are merged to obtain a third log, the third log includes a number field except for all fields in the first log or the second log, the number field is a sum of the number of the first log and the number of the second log, and finally, the log group is obtained according to the third log and logs which are not merged in the multiple logs.

For example, the multiple monitoring event logs are firewall logs, each firewall log includes fields such as a destination IP, a destination port, a source IP, a source port, a protocol type, a behavior, and time, and the behavior field is used to indicate whether to intercept the operation of the IP, where when a value of the behavior field is 1, it indicates that the interception operation is performed on the destination IP; the time field is used to indicate the moment of time at which the intercept operation is performed. In table 7, 3 firewall logs are included, and taking the first field as the time field as an example, the first log and the second log are first found from the plurality of firewall logs. In the first firewall log and the third firewall log, except for the difference of time fields, the values of other fields are the same, and the first firewall log and the third firewall log are considered to be capable of being merged, namely the first firewall log and the third firewall log are the first log and the second log. And then, combining the first log and the second log to obtain a new firewall log, as shown in table 8, where a value of the number field is a sum of the numbers of the first log and the second log, and in this example, if the sum of the numbers of the first log and the second log is 2, a value of the number field of the new firewall log is 2. In addition, it should be noted that the value of the time field in the new firewall log may be at a later time in the first log and the second log, for example, the value of the time field in the first log is 2019-10-1810:05, and the value of the time field in the second log is 2019-10-1810:15, and then the value of the time field in the new firewall log may be 2019-10-1810:15, or may also be set to a fixed time, for example, 2019-10-1810: 00, and the like, which is not limited herein.

TABLE 7

Destination IP	Destination port	Source IP	Source port	Type of protocol	Behavior	Time of day
							1.1.1.1	80	2.2.2.2	43202	Tcp	1	2019-10-18 10:05
1.1.1.2	443	3.3.3.3	23124	Tcp	1	2019-10-18 10:05
							1.1.1.1	80	2.2.2.2	43202	Tcp	1	2019-10-18 10:15

TABLE 8

Destination IP

Destination port

Source IP

Source port

Type of protocol

Behavior

Time of day

Number of times

1.1.1.1

80

2.2.2.2

43202

Tcp

1

2019-10-18 10:15

2

Then, a log group after merging processing is obtained according to the firewall log which is not subjected to merging processing in table 7 and the new firewall log in table 8. For example, the firewall log that is not merged in table 7 is the second firewall log, and then the second firewall log is combined with the new firewall log in table 8 to obtain the merged log group shown in table 7. In the log group shown in table 9, a number field is added to the firewall log that is not merged, and since the firewall log is not merged, the value of the number field of the firewall log is 1.

TABLE 9

Destination IP	Destination port	Source IP	Source port	Type of protocol	Behavior	Time of day	Number of times
								1.1.1.1	80	2.2.2.2	43202	Tcp	1	2019-10-18 10:15	2
1.1.1.2	443	3.3.3.3	23124	Tcp	1	2019-10-18 10:05	1

As can be seen from tables 7 and 9, the number of firewall logs is reduced after the merging process, and the processing delay of the ESPC can be reduced. In the above example, the first field is taken as one of the fields in the firewall log, and when the first field is a plurality of fields in the firewall log, the merging process is similar to the merging process for the traffic log, and details are not repeated here.

In addition, in the above example, the number of the first logs and the number of the second logs are all taken as 1, and in other examples, for example, as shown in table 10, the values of the fields except the flow value field in the first flow log, the third flow log and the fifth flow log are all the same, the first log may be considered as the first flow log, and the number of the second logs is 2, which are respectively the third flow log and the fifth flow log. It should be understood that "first" and "second" in this application are only used to distinguish different logs, and do not refer to a certain log specifically, and the number of the first log and the second log is not limited in this embodiment.

Watch 10

Destination IP	Source IP	Type of protocol	Destination port	Source port	Flow rate value
						1.1.1.1	2.2.2.1	Tcp	443	43201	200
1.1.1.2	2.2.2.2	Udp	53	32421	200
						1.1.1.1	2.2.2.1	Tcp	443	43201	300
1.1.1.2	3.3.3.4	Tcp	443	43242	300
						1.1.1.1	2.2.2.1	Tcp	443	43201	100

In the present application, the number of logs included in the log group after the merging process is related to the number of the first logs and the second logs, and the larger the sum of the number of the first logs and the number of the second logs is, the smaller the number of logs included in the log group after the merging process is. Because the values of the other fields except the first field in the first log and the second log are the same, the first log and the second log can be considered as repeated logs, and the more the repeated logs in the multiple logs are, the fewer the logs in the log group are after the merging processing.

In order to reduce the operation amount of the ESPC, the ESPC may merge the plurality of logs according to a preset period, for example, the preset period may be 1 hour or 2 hours. The merging process in each preset period may be similar to the foregoing, and will not be described herein again. In a possible implementation manner, the log group in the next period may be obtained by merging the log group obtained in the previous period with at least one log newly added in the period. For example, the log group obtained in the last period and the newly added logs in the period are merged into one data table, and then the merging processing is performed on a plurality of logs in the data table, so that the operation amount of the ESPC can be reduced.

Note that the merging processing in this embodiment is different from the merging processing in the related art.

The merging process in the related art is generally associated with statistical traffic. Specifically, in the merging process in the related art, firstly, the service dimension corresponding to the statistical service is predetermined, the service dimension may include a statistical object and a statistical result, and then the merging process is performed according to each service dimension that needs to be counted. For example, taking a plurality of logs shown in table 5 as an example, the merging process in the related art is as follows:

first, a statistical service is obtained in advance, for example, the statistical service is to count the flow trend of each IP per minute, and after merging the statistical service, a merged log group is obtained as shown in table 11.

TABLE 11

Destination IP	Time of day	Flow rate value
			1.1.1.1	2019/12/20 15:30:00	500
1.1.1.2	2019/12/20 15:30:00	500

Since the destination IP includes only 1.1.1.1 and 1.1.1.2 in the plurality of logs shown in table 5, the merged table 9 includes only two logs, and since the statistical service only concerns the traffic of each IP, the merged log group removes other fields in each log in table 5 and only retains the traffic value field, so that when the statistical service is changed, for example, the statistical service becomes the traffic trend of the statistical IP segment, which is (1.1.1.0/24), the log group shown in table 12 is obtained.

TABLE 12

Destination IP section	Time of day	Flow rate value
			1.1.1.0/24	2019/12/20 15:30:00	1000

It can be seen that, for the same data table, there are 10 service dimensions to be counted, and then the data needs to be merged for 10 times, and although the final counting speed is high, the merging times are many, and resources such as a CPU and a memory are consumed more. In addition, the merging process in the related art must know the service dimension to be counted in advance, if the statistical dimension cannot be determined, the merging cannot be performed, and in many usage scenarios, the service dimension corresponding to each statistical service cannot be known in advance, so the merging process in the related art cannot be applied to all scenarios.

As can be seen from the description in step 102, the first field used for merging is a field that is not to be a statistical object or a field that has a low probability of being a statistical object, which is contrary to the merging condition of the merging process in the related art. In addition, in the log group after merging processing in this embodiment, each log retains all the fields included in the log group. Taking the traffic log shown in table 2 as an example, each log in table 2 includes 5 fields of a destination IP, a destination port, a source IP, a source port, a protocol type, and a traffic value, and in the merged log group shown in table 4, each log still includes the 5 fields, that is, although merging processing is performed, the number of fields included in each log is not reduced, so that no matter what service dimension is statistical service, query can be performed in the log group after merging processing in this embodiment, and the method can be applied to all scenarios.

Step 103, acquiring a first statistical service, wherein the first statistical service comprises a first statistical object for performing report statistics on the plurality of logs.

In this embodiment, the first statistical service is any one statistical service received by the ESPC, and each statistical service includes a statistical object. For example, for the traffic log, the first statistical service may include counting the total traffic per minute for each IP, and the first statistical object is each destination IP; the first statistical service may be to count the total flow of a destination IP segment in one hour, and the first statistical object is the destination IP segment. For a monitoring event log, such as a firewall log, the first statistical service may be to count the total number of times that a certain specified IP is intercepted within one hour, and then the first statistical object is the specified IP; or the total number of times each IP is intercepted within one hour, the first statistical object is each destination IP. The specific contents of the first statistical service and the first statistical object are not limited herein.

And 104, inquiring the value of a field corresponding to the first statistical object in each log of the log group to obtain a log report corresponding to the first statistical service.

For example, taking the log group shown in table 4 as a traffic log recorded in one hour, the first statistical service is to count the total traffic of each IP in one hour. The ESPC queries the destination IP field in each log in the table 4, queries the first log to obtain a value of the destination IP field of 1.1.1.1, and queries a flow value corresponding to the log of 500, so that the first entry in the log report is generated as follows: destination IP: 1.1.1.1, flow value: 500. inquiring the second log and the third log to obtain a value of a target IP field of 1.1.1.2, and inquiring flow values respectively corresponding to the second log and the third log of 200 and 300, wherein a second table entry in a generated log report is as follows: destination IP: 1.1.1.2, the flow value is 500, resulting in the log report shown in table 13.

Watch 13

Destination IP	Flow rate value
		1.1.1.1	500
1.1.1.2	500

For another example, the log group shown in table 9 is the firewall log recorded in one hour, and the first statistical service is the first five names for counting the number of times of the blocked destination IP with the protocol type tcp in one day. In this case, the statistical object is a target IP, the statistical result is a frequency, the filter condition is that the value of the protocol type is TCP and the value of the behavior field is 1, the ESPC queries the target IP field in each log in the table 9, queries the first log to obtain a value of the target IP field of 1.1.1.1, queries a value of the frequency field corresponding to the log of 2, and generates a first entry in the log report as: destination IP: 1.1.1.1, times: 2. querying the second log to obtain a value of the destination IP field as 1.1.1.2, and querying a value of the number field corresponding to the second log as 1, if a second entry in the log report is generated as follows: destination IP: 1.1.1.2, the number of times is 1, thus obtaining the log report shown in table 14.

TABLE 14

Destination IP	Number of times
		1.1.1.1	2
1.1.1.2	1

In the above technical solution, when performing report statistics according to a plurality of stored logs, a field with a low probability of being a statistical object in a statistical service is first used as a merging point, the plurality of logs are merged, and then a field corresponding to the statistical object of a first statistical service is queried on the merged log according to a requirement of the first statistical service, so as to obtain a log report corresponding to the first statistical service. After the logs are merged, the number of the logs can be reduced, so that the number of the logs to be inquired can be reduced when the log report corresponding to the statistical service is obtained, and the smaller time delay of the statistical mode can be ensured. And because the number of the logs needing to be inquired can be reduced, under the condition of the log quantity of the same scale, the required hardware resources can be reduced, the cost can be reduced, and the statistical mode of the log report with low cost and smaller time delay is realized.

A second aspect of the present invention provides a statistical apparatus for log reports, which is shown in fig. 2 and is a schematic structural diagram of the statistical apparatus for log reports according to an embodiment of the present invention, where the apparatus includes:

a processing unit 201, configured to determine a first field used for merging the stored multiple logs, where the first field is a field whose probability of being a statistical object of the statistical traffic is lower than a threshold; merging the logs by using the first field as a merging point and using a residual field as a merging condition to obtain a log group, wherein the log group comprises at least one merged log, and the residual field is a field except the first field in the fields included in the logs;

an obtaining unit 202, configured to obtain a first statistical service, where the first statistical service includes a first statistical object that performs report statistics on the multiple logs;

the query unit 203 is configured to query a value of a field corresponding to the first statistical object in each log of the log group to obtain a log report corresponding to the first statistical service.

Since the statistical apparatus for a log report provided by the second aspect of the present invention is proposed under the same conception as the statistical method for a log report provided by the first aspect of the present invention, various variations and specific embodiments of the statistical method for a log report in the embodiment of fig. 1 are also applicable to the apparatus of the present embodiment, and through the foregoing detailed description of the statistical method for a log report, a person skilled in the art can clearly know the implementation process of the apparatus in the present embodiment, so for the brevity of the description, detailed description is not repeated here.

A third aspect of the present invention provides a statistical apparatus for log reports, and please refer to fig. 3, which is a structural diagram of the apparatus according to an embodiment of the present invention. As shown in fig. 3, the apparatus includes:

the processor 301 determines a first field for merging the stored logs, where the first field is a field with a probability of being a statistical object of the statistical traffic being lower than a threshold; merging the logs by using the first field as a merging point and using a residual field as a merging condition to obtain a log group, wherein the log group comprises at least one merged log, and the residual field is a field except the first field in the fields included in the logs;

then, the processor 301 obtains a first statistical service through the transceiver 302, where the first statistical service includes a first statistical object for performing report statistics on the plurality of logs;

the processor 301 queries a value of a field corresponding to the first statistical object in each log of the log group to obtain a log report corresponding to the first statistical service.

Optionally, the processor 301 may be a central processing unit, an Application Specific Integrated Circuit (ASIC), one or more Integrated circuits for controlling program execution, a hardware Circuit developed by using a Field Programmable Gate Array (FPGA), or a baseband processor.

Optionally, the processor 301 may include at least one processing core.

Optionally, the electronic device further includes a Memory 303, where the Memory 303 may include a Read Only Memory (ROM), a Random Access Memory (RAM), and a disk Memory. The memory 303 is used for storing data required by the processor 301 in operation. The number of the memories is one or more.

Since the statistical apparatus for a log report provided by the third aspect of the present invention is proposed under the same conception as the statistical method for a log report provided by the first aspect of the present invention, various variations and specific embodiments of the statistical method for a log report in the foregoing embodiment of fig. 1 are also applicable to the apparatus of this embodiment, and through the foregoing detailed description of the statistical method for a log report, a person skilled in the art can clearly know the implementation process of the apparatus in this embodiment, so for the brevity of the description, detailed description is not repeated here.

A fourth aspect of the present invention provides a computer apparatus, comprising:

at least one processor, and

wherein the memory stores instructions executable by the at least one processor, and the at least one processor performs the method of the embodiment shown in fig. 1 using the communication interface by executing the instructions stored by the memory.

A fifth aspect of the present invention provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores computer instructions which, when run on a computer, cause the computer to perform the method in the embodiment shown in fig. 1.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, and the like) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.

Claims

1. A statistical method of log reports is characterized by comprising the following steps:

2. The method of claim 1, wherein the merging the plurality of logs to obtain the log group, the merging being performed on the plurality of logs with the first field as a merging point and the remaining fields as merging conditions, and the method including:

3. The method of claim 1, wherein the merging the plurality of logs to obtain the log group comprises:

4. The method according to claim 2 or 3, wherein the larger the sum of the number of the first logs and the second logs, the smaller the number of logs included in the log group.

5. The method according to any one of claims 1-4, wherein merging the plurality of logs to obtain a log group comprises:

6. A statistical device of log reports is characterized by comprising:

7. The apparatus according to claim 6, wherein the plurality of logs are all traffic logs, the first field is a traffic field, and the processing unit merges the plurality of logs by using the first field as a merging point and using the remaining fields as a merging condition, to obtain a log group, and is specifically configured to:

8. The apparatus according to claim 6, wherein the logs are all monitoring event logs, the first field is a time field, the processing unit merges the logs by using the first field as a merging point and using the remaining fields as a merging condition, so as to obtain a log group, and is specifically configured to:

9. The apparatus according to claim 7 or 8, wherein the larger the sum of the number of the first logs and the second logs, the smaller the number of logs included in the log group.

10. The apparatus according to any one of claims 6 to 9, wherein the processing unit merges the plurality of logs to obtain a log group, and is specifically configured to:

11. A computer device, the computer device comprising:

at least one processor, and

wherein the memory stores instructions executable by the at least one processor, the at least one processor performing the method of any one of claims 1-5 with the communications interface by executing the instructions stored by the memory.

12. A computer-readable storage medium having stored thereon computer instructions which, when executed on a computer, cause the computer to perform the method of any one of claims 1-5.