CN115883317A

CN115883317A - Log data processing method and device, electronic equipment and storage medium

Info

Publication number: CN115883317A
Application number: CN202211574693.6A
Authority: CN
Inventors: 任亚娟
Original assignee: Neusoft Corp
Current assignee: Neusoft Corp
Priority date: 2022-12-08
Filing date: 2022-12-08
Publication date: 2023-03-31

Abstract

The disclosure relates to a log data processing method and device, an electronic device and a storage medium. The method comprises the following steps: acquiring log data corresponding to network equipment; the log data comprises a log recorded with a sending end identifier of sending end equipment, wherein the sending end equipment is equipment for sending a message to network equipment or sending a message flowing through the network equipment; determining a target log type and a sending end identifier corresponding to log data; determining an abnormal analysis index corresponding to the sending terminal identification according to the target log type; and executing an exception handling operation according to the sending end identifier under the condition that the exception analysis index is greater than or equal to the preset index threshold value. Therefore, the log data can be analyzed and counted in the network equipment, the processing efficiency of the log data is improved, the processing efficiency of the abnormal sending terminal identification is also improved, and in addition, the cost can be reduced because other equipment is not needed.

Description

Log data processing method and device, electronic equipment and storage medium

Technical Field

The present disclosure relates to the field of computer technologies, and in particular, to a log data processing method and apparatus, an electronic device, and a storage medium.

Background

With the development and popularization of computer network technology, a large website system is often accessed by malicious IPs, bandwidth loss is caused, and normal access of a website is affected. The firewall technology has the functions of timely finding and processing the possible safety risk, data transmission and other problems in the running of computer network, and the processing measures include isolation and protection, and recording and detecting the operation in the computer network safety to ensure the running safety of computer network, ensure the integrity of user data and information and provide better and safer computer network use experience for users.

However, in the related art, it is often necessary to manually process the malicious IP according to the log data of the firewall, which is inefficient.

Disclosure of Invention

To overcome the above problems in the related art, the present disclosure provides a log data processing method, apparatus, electronic device, and storage medium.

According to a first aspect of embodiments of the present disclosure, there is provided a log data processing method, including:

acquiring log data generated by network equipment; the log data comprises a log recorded with a sending end identifier of sending end equipment, and the sending end equipment is equipment for sending a message to the network equipment or sending a message flowing through the network equipment;

determining a target log type and the sending end identification corresponding to the log data;

determining an abnormal analysis index corresponding to the sending end identifier according to the target log type;

and determining the sending end identification as an abnormal sending end identification under the condition that the abnormal analysis index is greater than or equal to a preset index threshold value.

In some embodiments, the determining, according to the target log type, an anomaly analysis indicator corresponding to the sender identifier includes:

determining an abnormal analysis adjustment parameter corresponding to the target log type according to the log parameter corresponding relation; the log parameter corresponding relation comprises a corresponding relation between the target log type and the abnormal analysis adjustment parameter;

and determining an abnormal analysis index corresponding to the sending end identifier according to the abnormal analysis adjustment parameter.

In some embodiments, the determining, according to the anomaly analysis adjustment parameter, the anomaly analysis indicator corresponding to the sender identifier includes:

under the condition that a historical abnormal analysis index corresponding to the sending end identification is obtained, determining the abnormal analysis index according to the historical abnormal analysis index and the abnormal analysis adjustment parameter, and taking the abnormal analysis index as a new historical abnormal analysis index; or,

and under the condition that the historical abnormal analysis index corresponding to the sending end identifier is not obtained, determining the abnormal analysis index corresponding to the sending end identifier according to the abnormal analysis adjustment parameter, and taking the abnormal analysis index as the historical abnormal analysis index corresponding to the sending end identifier.

In some embodiments, the log parameter correspondence is generated by:

determining a plurality of candidate log types; the candidate log type comprises the target log type;

acquiring a first data volume generated by each candidate log type in a first time period; the first time period is a historical time period before the current moment;

calculating to obtain a first anomaly analysis adjustment parameter corresponding to each candidate log type according to the first data volume; the more the first data volume is, the smaller the first anomaly analysis adjustment parameter obtained through calculation is;

and taking the first abnormal analysis adjustment parameter as an abnormal analysis adjustment parameter corresponding to the candidate log type to obtain the corresponding relation of the log parameters.

In some embodiments, the log parameter correspondence is updated by:

acquiring the number of logs counted in a second time period and a third data volume in a third time period for each candidate log type; the second time period and the third time period are both time periods before the current moment, and the third time period comprises the second time period;

determining a second weight corresponding to the third data volume and a first weight corresponding to the log number, wherein the first weight is greater than or equal to the second weight;

calculating to obtain a second anomaly analysis adjustment parameter corresponding to each candidate log type according to the third data volume, the second weight, the log number and the first weight;

and taking the second abnormal analysis adjustment parameter as an abnormal analysis adjustment parameter corresponding to the candidate log type to obtain the log parameter corresponding relation.

In some embodiments, the method further comprises

Acquiring the current quantity of the sending end record information currently stored by the network equipment; each sending end recording information comprises a sending end identification, a recording time and a historical abnormal analysis index corresponding to the sending end identification, wherein the recording time is the time when the sending end recording information is stored to the network equipment for the first time;

determining a target duration according to the current number and a preset number threshold when the current number is greater than or equal to the preset number threshold;

and deleting the sending end recording information with the storage duration being greater than or equal to the target duration from the network equipment, wherein the storage duration is used for representing the time difference between the recording time and the current time.

In some embodiments, the determining the target log type and the sender identity corresponding to the log data includes:

according to a preset type parameter, performing text analysis on the log data, and determining a corresponding target log type in the log data;

and according to a preset regular expression, performing text analysis on the log data to obtain the sending end identifier.

In some embodiments, the method further comprises:

and executing an exception handling operation according to the sending end identifier under the condition that the exception analysis index is greater than or equal to a preset index threshold value.

According to a second aspect of the embodiments of the present disclosure, there is provided a log data processing apparatus, the apparatus including:

the log acquisition module is used for acquiring log data generated by the network equipment; the log data comprises a log recorded with a sending end identifier of sending end equipment, and the sending end equipment is equipment for sending a message to the network equipment or sending a message flowing through the network equipment;

the log processing module is used for determining a target log type and the sending end identification corresponding to the log data;

the index analysis module is used for determining an abnormal analysis index corresponding to the sending end identifier according to the target log type;

and the exception handling module is used for executing exception handling operation according to the sending end identifier under the condition that the exception analysis index is greater than or equal to a preset index threshold value.

In some embodiments, the index analysis module is configured to determine an anomaly analysis adjustment parameter corresponding to the target log type according to a log parameter correspondence; the log parameter corresponding relation comprises a corresponding relation between the target log type and the abnormal analysis adjustment parameter; and determining an abnormal analysis index corresponding to the sending end identifier according to the abnormal analysis adjustment parameter.

In some embodiments, the index analysis module is configured to, when a historical abnormal analysis index corresponding to the sender identifier is obtained, determine the abnormal analysis index according to the historical abnormal analysis index and the abnormal analysis adjustment parameter, and use the abnormal analysis index as a new historical abnormal analysis index; or, under the condition that the historical abnormal analysis index corresponding to the sending end identifier is not obtained, determining the abnormal analysis index corresponding to the sending end identifier according to the abnormal analysis adjustment parameter, and using the abnormal analysis index as the historical abnormal analysis index corresponding to the sending end identifier.

In some embodiments, the apparatus further comprises:

an initialization module for determining a plurality of candidate log types; the candidate log type comprises the target log type; acquiring a first data volume generated by each candidate log type in a first time period; the first time period is a historical time period before the current moment; calculating to obtain a first anomaly analysis adjustment parameter corresponding to each candidate log type according to the first data volume; the more the first data volume is, the smaller the first anomaly analysis adjustment parameter obtained through calculation is; and taking the first abnormal analysis adjustment parameter as an abnormal analysis adjustment parameter corresponding to the candidate log type to obtain the corresponding relation of the log parameters.

In some embodiments, the apparatus further comprises:

the dynamic planning module is used for acquiring the log quantity counted in the second time period and the third data volume in the third time period of each candidate log type; the second time period and the third time period are both time periods before the current moment, and the third time period comprises the second time period; determining a second weight corresponding to the third data volume and a first weight corresponding to the log quantity, wherein the first weight is greater than or equal to the second weight; calculating to obtain a second anomaly analysis adjustment parameter corresponding to each candidate log type according to the third data volume, the second weight, the log quantity and the first weight; and taking the second abnormal analysis adjustment parameter as an abnormal analysis adjustment parameter corresponding to the candidate log type to obtain the log parameter corresponding relation.

In some embodiments, the apparatus further comprises:

the data eliminating module is used for acquiring the current quantity of the sending end record information currently stored by the network equipment; each sending end record message comprises a sending end identifier, a record time and a historical abnormal analysis index corresponding to the sending end identifier, wherein the record time is the time when the sending end record message is stored to the network equipment for the first time; determining a target duration according to the current number and a preset number threshold under the condition that the current number is greater than or equal to the preset number threshold; and deleting the sending end recording information with the storage duration being greater than or equal to the target duration from the network equipment, wherein the storage duration is used for representing the time difference between the recording time and the current time.

In some embodiments, the log processing module is configured to perform text analysis on the log data according to a preset type parameter, and determine a corresponding target log type in the log data; and according to a preset regular expression, performing text analysis on the log data to obtain the sending end identifier.

According to a third aspect of an embodiment of the present disclosure, there is provided an electronic apparatus including: a memory having a computer program stored thereon; a processor for executing the computer program in the memory to implement the steps of the method of the first aspect of the disclosure.

According to a fourth aspect of embodiments of the present disclosure, there is provided a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the method of the first aspect of the present disclosure.

The technical scheme provided by the embodiment of the disclosure can have the following beneficial effects: acquiring log data corresponding to network equipment; the log data may include a log in which a sending end identifier of sending end equipment is recorded, where the sending end equipment is equipment that sends a message to network equipment or sends a message flowing through the network equipment; determining a target log type and a sending end identifier corresponding to log data; determining an abnormal analysis index corresponding to the sending terminal identification according to the target log type; and executing an exception handling operation according to the sending end identifier under the condition that the exception analysis index is greater than or equal to the preset index threshold value. Therefore, the log data can be analyzed and counted in the network equipment, the processing efficiency of the log data is improved, the processing efficiency of the abnormal sending terminal identification is also improved, and in addition, the cost can be reduced because other equipment is not needed.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure.

Fig. 1 is a flowchart illustrating a log data processing method according to an exemplary embodiment.

Fig. 2 is a flowchart illustrating a method of generating log parameter correspondences, according to an example embodiment.

Fig. 3 is a flowchart illustrating a method of updating log parameter correspondences, according to an example embodiment.

Fig. 4 is a block diagram illustrating a log data processing apparatus according to an example embodiment.

Fig. 5 is a block diagram illustrating another log data processing apparatus according to an example embodiment.

FIG. 6 is a block diagram illustrating an electronic device in accordance with an example embodiment.

Detailed Description

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the exemplary embodiments below do not represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

It should be noted that all actions of acquiring signals, information or data in the present disclosure are performed under the premise of complying with the corresponding data protection regulation policy of the country of the location and obtaining the authorization given by the owner of the corresponding device.

In the description of the present disclosure, terms such as "first," "second," and the like are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. In addition, in the description with reference to the drawings, the same reference numerals in different drawings denote the same elements, but not explained to the contrary.

In the description of the present disclosure, unless otherwise indicated, "plurality" means two or more, and other terms are similar; "at least one item", "one or more item", or similar expressions, refers to any combination of these item(s), including any combination of single item(s) or plural item(s). For example, at least one item(s) a, may represent any number a; as another example, one or more of a, b, and c may represent: a, b, c, a-b, a-c, b-c, or a-b-c, wherein a, b, c can be single or multiple; "and/or" is an association describing an associated object, meaning that there may be three relationships, e.g., a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone, wherein A and B can be singular or plural.

Although operations or steps may be described in a particular order in the drawings in the embodiments of the disclosure, they should not be construed as requiring that such operations or steps be performed in the particular order shown or in serial order, or that all illustrated operations or steps be performed, to achieve desirable results. In embodiments of the present disclosure, these operations or steps may be performed in series; these operations or steps may also be performed in parallel; some of these operations or steps may also be performed.

The present disclosure is described below with reference to specific examples.

First, an application scenario of the present disclosure will be explained. The present disclosure may be applied to an abnormal data processing scenario of a firewall or other network browsing device. In order to process malicious IPs, a network device such as a firewall may generate a large amount of log data, manually analyze the log data to find a problem IP, and make a policy for processing the problem IP, for example, block or add the problem IP configuration policy to a blacklist.

In one implementation manner of the present disclosure, log data of a network device such as a firewall may be sent to a destination device, then the number of the problem IPs is counted by analyzing a history log through the destination device, and then a policy is configured (the IP configuration policy is blocked or added to a blacklist) on the network device such as the firewall through a manual or automatic manner. The method not only needs other equipment and depends on manual processing, but also has low efficiency and low real-time property.

Fig. 1 is a flowchart illustrating a log data processing method according to an exemplary embodiment. The method may be applied to network devices, which may include firewalls, routers, computers or servers, among other devices. As shown in fig. 1, the method may include:

s101, obtaining log data generated by the network equipment.

The log data may include a log in which a sending end identifier of sending end equipment is recorded, and the sending end equipment may be equipment that sends a message to network equipment or sends a message flowing through the network equipment.

For example, the destination IP address of the packet sent by the sending end device may be an IP address of the network device, an IP address that can be reached by flowing through the network device, or other IP addresses, which is not limited in this disclosure.

The sender identification may include one or more of an IP address, a port number, a MAC address, and a device identification of the sender device.

In some embodiments, the sender identification may include an IP address of the sender device, for example, the source IP in a message sent by the sender device.

In other embodiments, the sender id may include a Media Access Control (MAC) address of the sender device.

In some other embodiments, the sender identification may include the source IP and source port number in the message sent by the sender device.

It should be noted that, when processing a message sent by a sending end device, a network device may directly generate log data, or may generate log data when the message satisfies a condition. For example, the log data may be generated in the case where the network device determines that the message is an abnormal message (e.g., an offensive message or a message determined to be unsafe).

It should be further noted that, in the multiple log data generated by the network device, part of the log data may record the sender identifier, and part of the log data may not record the sender identifier, which is not limited in this disclosure.

S102, determining a target log type and a sending end identification corresponding to the log data.

In some embodiments, the log data may be subjected to text parsing according to a preset type parameter, and a corresponding target log type in the log data is determined.

For example, the preset type parameter may include "key information of all log types that need to be parsed" that is set in advance, and the preset type parameter may be a key text in log data, from which the target log type may be determined. For example: the preset type parameter may include a key text "traffic attack", and the target log type corresponding to the log data having the preset type parameter "traffic attack" may be a traffic attack log.

In some embodiments, the log data may be subjected to text parsing according to a preset regular expression, so as to obtain the sender identifier. The preset regular expression may be a regular expression for extracting a sender identity (e.g., IP or MAC).

In some embodiments, the step S102 may be performed by a log processing module, where an input of the log processing module may include log data, and an output may include a target log type and a sender identifier corresponding to the log data.

In some embodiments, the log data of the target log type and the sender identifier obtained by the analysis may be used as abnormal data for subsequent analysis.

In other embodiments, the log data meeting the preset abnormal condition may be selected from the log data of the target log type and the sender identifier obtained by the analysis as abnormal data for subsequent analysis. For example, the preset exception condition may include that the target log type is a preset exception type, that is, log data whose target log type is a preset exception type may be taken as exception data. The preset abnormal type is a plurality of preset types.

S103, determining an abnormal analysis index corresponding to the sending end identification according to the target log type.

The anomaly analysis index can be used for representing the anomaly information quantity or the anomaly influence degree of the transmitting end identification.

In some embodiments, the anomaly analysis index may be determined based on the amount of data of the target log type. The data amount may be the number of log data corresponding to the target log type, or may be the size of the storage space occupied by the log data corresponding to the target log type.

In other embodiments, an anomaly analysis adjustment parameter corresponding to a target log type may be determined first; and determining an abnormal analysis index corresponding to the sending end identifier according to the abnormal analysis adjustment parameter.

The anomaly analysis adjustment parameter may also be referred to as a log score, and may be used to represent an anomaly information amount or an anomaly influence degree of each piece of log data corresponding to the target log type, where the larger the anomaly analysis adjustment parameter is, the larger an anomaly information amount generated by one piece of log data that can represent the target log type is, the higher the anomaly influence degree is.

It should be noted that the log data in this embodiment may be one piece of log data or a plurality of pieces of log data, and if there are a plurality of pieces of log data, the abnormality analysis adjustment parameter may be obtained by multiplying the abnormality analysis adjustment parameter by the number of pieces of log data to accumulate the plurality of pieces of log data.

The anomaly analysis adjustment parameters corresponding to different target log types can be the same or different. For example, the anomaly analysis adjustment parameter corresponding to the target log type may be determined according to a log parameter corresponding relationship, where the log parameter corresponding relationship may include a corresponding relationship between the target log type and the anomaly analysis adjustment parameter.

The method for determining the anomaly analysis index corresponding to the sender identifier according to the anomaly analysis adjustment parameter may include any one of the following:

in the first mode, under the condition that the historical abnormal analysis index corresponding to the sending end identifier is obtained, the abnormal analysis index is determined according to the historical abnormal analysis index and the abnormal analysis adjustment parameter, and the abnormal analysis index is used as a new historical abnormal analysis index.

Illustratively, the log statistics may be stored in the network device. The log statistics may include one or more historical sender identifiers and historical anomaly analysis indicators corresponding to each historical sender identifier.

Therefore, whether the historical sending end identification in the log statistical information comprises the sending end identification can be determined firstly, if yes, the historical abnormal analysis index corresponding to the sending end identification can be obtained, and the abnormal analysis index corresponding to the sending end identification can be obtained through calculation according to the historical abnormal analysis index and the abnormal analysis adjustment parameter. For example, the sum of the historical abnormality analysis index and the abnormality analysis adjustment parameter may be used as the abnormality analysis index, and the abnormality analysis index may be stored as a new historical abnormality analysis index in the log statistical information.

And in the second mode, under the condition that the historical abnormal analysis index corresponding to the sending end identifier is not obtained, determining the abnormal analysis index corresponding to the sending end identifier according to the abnormal analysis adjustment parameter, and taking the abnormal analysis index as the historical abnormal analysis index corresponding to the sending end identifier.

Similarly, it may be determined whether the historical sender identifier in the log statistical information includes the sender identifier, and if not, it may be determined that the historical abnormal analysis index corresponding to the sender identifier is not obtained, so that the abnormal analysis index corresponding to the sender identifier may be determined according to the abnormal analysis adjustment parameter, for example, the abnormal analysis adjustment parameter may be used as the abnormal analysis index, or the abnormal analysis adjustment parameter may be multiplied by a preset coefficient to obtain the abnormal analysis index.

Further, the anomaly analysis index can be used as a historical anomaly analysis index corresponding to the sender identifier and stored in the log statistical information.

Still further, the current time may be used as the first recording time of the sender identifier, and the first recording time is stored in the log statistical information.

And S104, executing an exception handling operation according to the sending end identifier under the condition that the exception analysis index is greater than or equal to the preset index threshold value.

The preset index threshold may be any preset value.

In some embodiments, the exception handling operation may include: and determining the sender identifier as an abnormal sender identifier.

When the anomaly analysis index of the sender id is greater than or equal to the preset index threshold, it may be determined that the amount of anomaly information or the anomaly influence accumulated by the sender id has exceeded the preset degree, and therefore, the sender id may be determined as an anomalous sender id.

For example, in the case that the sender is identified as the source IP, the target source IP may be determined as an abnormal IP.

Further, in the case that the sender id is determined to be an abnormal sender id, an abnormal prompt message including the sender id may be presented to the user. The exception prompt information can be used for prompting a user to perform exception handling operation on the sending end identifier.

In other embodiments, the exception handling operation may include: and executing a target instruction according to the sender identifier, wherein the target instruction can be an instruction generated according to a preset basic instruction and the sender identifier.

For example, the preset basic instruction may be a basic instruction for performing exception handling, and after the sender identifier and the preset basic instruction are combined, an executable target instruction may be obtained, and the exception handling operation may be implemented by executing the target instruction.

For example, the sender id may be a source IP, the preset basic instruction may be a basic command for adding an IP to a blacklist or configuring a policy blocking IP, and after combining the source IP and the basic command, an executable command (target instruction) may be obtained, and the command is executed, so that an operation of adding the source IP to the blacklist or configuring the policy blocking IP may be implemented.

Therefore, the method can further improve the timeliness and save the labor cost by automatically executing the exception handling operation.

By adopting the method, the log data corresponding to the network equipment is obtained; the log data may include a log in which a sending end identifier of sending end equipment is recorded, where the sending end equipment is equipment that sends a message to network equipment or sends a message flowing through the network equipment; determining a target log type and a sending end identifier corresponding to log data; determining an abnormal analysis index corresponding to the sending terminal identification according to the target log type; and executing an exception handling operation according to the sending end identifier under the condition that the exception analysis index is greater than or equal to the preset index threshold value. Therefore, the log data can be analyzed and counted in the network equipment, the processing efficiency of the log data is improved, the processing efficiency of the abnormal sending terminal identification is also improved, and in addition, the cost can be reduced because other equipment is not needed.

In some embodiments of the present disclosure, the log parameter correspondence may be a preset correspondence, for example, the correspondence may be an anomaly analysis adjustment parameter preset by a user according to experience for each candidate log type, where the candidate log type may include a target log type.

In other embodiments of the present disclosure, the log parameter correspondence may be generated based on historical statistical data.

Fig. 2 is a flowchart illustrating a method of generating log parameter correspondences, according to an example embodiment. As shown in fig. 2, the method may include:

s201, determining multiple candidate log types.

The candidate log type may include the target log type. Illustratively, the candidate log type may be any log type set in advance.

S202, acquiring a first data volume generated in a first time period by each candidate log type.

Wherein the first time period may be a historical time period before the current time. For example, the time period may be a complete historical time period from the power-on of the network device to the current time, or may be a historical time period within a first preset time period before the current time, where the first preset time period may be any preset time period, for example, 1 day, 10 days, 1 month, 2 months, 1 quarter, or 1 year.

The first data amount may be the number of the log data counted by the candidate log type in the first time period, or may also be the size of the storage space occupied by the log data counted by the candidate log type in the first time period, which is not limited in this disclosure.

And S203, calculating to obtain a first anomaly analysis adjustment parameter corresponding to each candidate log type according to the first data volume.

The more the first data amount is, the smaller the calculated first anomaly analysis adjustment parameter is.

For example, assume that n candidate log types are determined, each of which generates a first amount of data in a first time period [ d ] ₁ ,d ₂ ,d ₃ ,…,d _n ]First, a first ratio [ p ] of each candidate log type can be calculated according to the following formula (1) ₁ ,p ₂ ,p ₃ ,…,p _n ]：

Wherein p is _i First ratio representing i-th candidate Log type, d _i The log data size is a first data size generated by the ith candidate log type in a first time period, n represents the total number of the candidate log types, and the value of i can be 1 to n.

In some embodiments, the inverse of the first ratio may be used as the first anomaly analysis adjustment parameter.

In other embodiments, a difference obtained by subtracting the first ratio from 1 may be used as the first anomaly analysis adjustment parameter.

In some other embodiments, the first anomaly analysis adjustment parameter corresponding to each candidate log type may be calculated according to the following formula (2):

wherein, b _i A first anomaly analysis adjustment parameter, p, representing the correspondence of the ith candidate log type _i The log type is represented by a first proportion of the ith candidate log type, k represents a first preset optimization base number, and the first preset optimization base number can be any preset value larger than 1.

In this way, the first anomaly analysis adjustment parameter corresponding to each candidate log type can be calculated according to the first data volume and the first preset optimization base number.

And S204, taking the first abnormal analysis adjustment parameter as an abnormal analysis adjustment parameter corresponding to the candidate log type to obtain a log parameter corresponding relation.

Thus, the log parameter correspondence can be generated in the above manner.

In some embodiments, the steps of generating the log parameter correspondence relationship shown in S201 to S204 above may be performed by an initialization module.

In some embodiments of the present disclosure, the log parameter correspondence may also be updated through an updating step. Fig. 3 is a flowchart illustrating a method of updating log parameter correspondences, according to an example embodiment. As shown in fig. 3, the updating method may include:

s301, acquiring the number of logs counted in the second time period and the third data volume in the third time period for each candidate log type.

The second time period and the third time period are both time periods before the current time, and the third time period comprises the second time period.

For example, the second time period may be a historical time period within a second preset time period before the current time, and the second preset time period may be any preset time period, such as 1 day, 10 days, 1 month, or 2 months.

The third time period may be a historical time period prior to the current time. For example, the time period may be a complete historical time period from the power-on start of the network device to the current time, or may be a historical time period within a third preset time period before the current time, where the third preset time period may be any preset time period, for example, 10 days, 1 month, 2 months, 1 quarter, or 1 year.

In some embodiments, the third predetermined period of time may be greater than the second predetermined period of time.

The log quantity may be the number of the log data counted by the candidate log type in the second time period, or the size of the storage space occupied by the log data newly counted by the candidate log type in the second time period; the third data amount may be the number of the log data counted by the candidate log type in the third time period, or may also be the size of a storage space occupied by the log data newly counted by the candidate log type in the third time period, which is not limited in this disclosure.

In some embodiments, the number of logs in the second time period is the number of logs obtained from the log type and the sender identifier counted to each candidate log type, and the third data volume in the third time period may include the data volume of log data obtained from the log type, for example, the third data volume may include the total data volume of log data obtained from the log type and the sender identifier, and log data obtained from the log type but not obtained from the sender identifier.

It should be noted that, the obtaining manner of the third data amount in the third time period may be the same as the obtaining manner of the first data amount in the first time period, and specifically refer to the description in the foregoing embodiment of the disclosure, and details are not repeated here.

S302, determining a first weight corresponding to the number of logs and a second weight corresponding to the third data volume.

Wherein the first weight may be greater than or equal to the second weight.

In some embodiments, the first weight and the second weight may be any parameters set in advance. In one implementation, the sum of the first weight and the second weight is 1, and the first weight may be greater than or equal to the second weight. For example, the first weight may be 0.6, the second weight may be 0.4; for another example, the first weight may be 0.8, and the second weight may be 0.2.

In other embodiments, the first weight may be calculated according to the number of logs, and the second weight may be calculated according to the third data amount.

In some implementations, the second weight is calculated according to the third amount of data as follows:

assuming that n candidate journal types are determined, the third data amount generated in the third time period for each candidate journal type is [ e ], respectively ₁ ,e ₂ ,e ₃ ,…,e _n ]The first information value of each candidate log type can be first calculated according to the following formula (3):

e′ _i ＝Max(e _x )-e _i +Δ(x＝1，2，3，…，n) (3)

wherein, e' _i Representing a first information value, max (e), corresponding to the i-th candidate log type _x ) Represents the maximum value of the n third data volumes, e _i Representing a third data volume corresponding to the ith candidate log type, wherein delta represents a preset fault-tolerant parameter, and the preset fault-tolerant parameter can be a preset smaller positive number to avoid e' _i Is 0, for example, the preset fault tolerance parameter may be 0.01 or 0.001, etc.

Then, the second weight k2 may be calculated according to the following equation (4):

wherein k2 represents a second weight, max (e' _i ) Representing a maximum of the first information values of the n candidate journal typesValue of,

a sum value representing a first information value of the n candidate journal types.

In this way, the second weight can be calculated.

In some implementations, the first weight is calculated from the number of logs as follows:

similarly, assuming that n candidate journal types are determined, the number of the journals counted in the second time period for each candidate journal type is [ c ₁ ,c ₂ ,c ₃ ,…,c _n ]The second information value of each candidate log type may be first calculated according to the following equation (5):

c′ _i ＝Max(c _x )-c _i +Δ(x＝1，2，3，…，n) (5)

wherein, c' _i Representing a second information value, max (c), corresponding to the i-th candidate log type _x ) Represents the maximum of n log numbers, c _i Representing the number of logs corresponding to the ith candidate log type, wherein delta represents a preset fault-tolerant parameter, and the preset fault-tolerant parameter can be a preset small positive number avoiding e' _i Is 0, for example, the preset fault tolerance parameter may be 0.01 or 0.001, etc.

Then, the first weight k1 may be calculated according to the following equation (6):

wherein k1 represents a first weight, max (c' _i ) Represents the maximum value of the second information values of the n candidate journal types,

a sum value representing a second information value of the n candidate journal types.

In this way, the first weight k1 can be calculated.

Further, if the calculated first weight is smaller than the second weight, the first weight may be set to a value greater than or equal to the second weight, and for example, the calculated second weight may be set to the value of the first weight, so that the weight of data closer to the current time can be secured to be greater than the weight of data farther away.

In some embodiments, after the first weight is calculated, the log number may be cleared to restart the statistics.

And S303, calculating to obtain a second anomaly analysis adjustment parameter corresponding to each candidate log type according to the number of the logs, the third data volume, the first weight and the second weight.

In some embodiments, a third anomaly analysis adjustment parameter corresponding to each candidate log type may be obtained by first calculating according to the number of logs, and a fourth anomaly analysis adjustment parameter corresponding to each candidate log type may be obtained by calculating according to the third data amount; and then, calculating to obtain a second abnormal analysis adjusting parameter according to the third abnormal analysis adjusting parameter, the fourth abnormal analysis adjusting parameter, the second weight and the first weight.

For example, assume that there are n candidate log types, and each candidate log type generates a third amount of data within a third time period [ e [ ] ₁ ,e ₂ ,e ₃ ,…,e _n ]The third ratio [ r ] for each candidate log type may be first calculated according to the following equation (7) ₁ ,r ₂ ,r ₃ ,…,r _n ]：

Wherein r is _i Third ratio representing i-th candidate Log type, e _i And the data amount of the ith candidate log type generated in the third time period is represented, n represents the total number of the candidate log types, and the value of i can be 1 to n.

Then, a fourth anomaly analysis adjustment parameter may be calculated based on the third ratio.

In some embodiments, the inverse of the third ratio may be used as the fourth anomaly analysis adjustment parameter.

In other embodiments, a difference obtained by subtracting the third ratio from 1 may be used as the fourth anomaly analysis adjustment parameter.

In some other embodiments, the fourth anomaly analysis adjustment parameter corresponding to each candidate log type may be calculated according to the following formula (8):

wherein w _i A fourth anomaly analysis adjustment parameter, r, corresponding to the ith candidate log type _i And k represents a first preset optimization base number, wherein the first preset optimization base number can be any preset value larger than 1.

In this way, the fourth abnormality analysis adjustment parameter can be calculated.

Similarly, assuming that n candidate journal types are determined, the number of the journals counted in the second time period for each candidate journal type is [ c ₁ ,c ₂ ,c ₃ ,…,c _n ]The second ratio [ t ] for each candidate log type may be first calculated according to equation (9) below ₁ ,t ₂ ,t ₃ ,…,t _n ]：

Wherein, t _i Second ratio representing i-th candidate Log type, c _i The log number counted in the second time period by the ith candidate log type is represented, n represents the total number of the candidate log types, and the value of i can be 1 to n.

A third anomaly analysis adjustment parameter may then be calculated from the second ratio.

In some embodiments, the inverse of the second ratio may be used as the third anomaly analysis tuning parameter.

In other embodiments, a difference obtained by subtracting the second ratio from 1 may be used as the third anomaly analysis adjustment parameter.

In some other embodiments, the third anomaly analysis adjustment parameter corresponding to each candidate log type may be calculated according to the following formula (10):

wherein u is _i A third anomaly analysis adjustment parameter, t, corresponding to the ith candidate log type _i And w represents a second preset optimization base number, wherein the second preset optimization base number can be any preset value greater than 1, and the second preset optimization base number and the first preset optimization base number can be the same or different.

In this way, the third anomaly analysis adjustment parameter can be calculated.

Further, the second anomaly analysis adjustment parameter may be calculated by the following equation (11):

s _i ＝(k1*w _i +k2*u _i )/(k1+k2) (i＝1，2，3，…，n) (11)

wherein s is _i Representing a second anomaly analysis adjustment parameter corresponding to the ith candidate log type, k1 representing a second weight, k2 representing a first weight, w _i A fourth anomaly analysis adjustment parameter u representing the correspondence of the ith candidate log type _i And indicating a third anomaly analysis adjustment parameter corresponding to the ith candidate log type.

In this way, the second anomaly analysis adjustment parameter can be calculated.

S304, taking the second abnormal analysis adjustment parameter as an abnormal analysis adjustment parameter corresponding to the candidate log type to obtain a log parameter corresponding relation.

By the method, the corresponding relation of the log parameters can be generated or updated.

In some embodiments of the present disclosure, the steps of S301 to S304 may be performed periodically, for example, a preset period may be set, and the preset period may be performed once every preset period, and the preset period may be any preset time length.

In some implementations, the second time period may be a time period within the current cycle, that is, a second preset time period for determining the second time period is equal to the time period of the preset cycle.

In some embodiments, the steps of updating the log parameter correspondence shown in S301 to S304 may be performed by the dynamic programming module.

In some embodiments of the present disclosure, a network device may store one or more pieces of sender record information, where each piece of sender record information may include a sender identifier, a record time, and a historical anomaly analysis indicator corresponding to the sender identifier, and the record time is a time when the sender record information is first stored in the network device. The sender id recorded by different sender record information may be different. The sending-end record information may further include a record time, and the record time may be used to represent a time at which the sending-end record information is first stored in the network device. For example, the sender id may be obtained by first parsing, and the time when the sender records information may be recorded.

In some embodiments, sending end record information currently stored by the network device may be reasonably removed to reduce resource occupation.

In some implementation manners, a current number of sending-end record information currently stored by the network device may be obtained, and when the current number is greater than or equal to a preset number threshold, a target duration is determined according to the current number and the preset number threshold; and deleting the sending end record information with the storage duration being greater than or equal to the target duration from the network equipment.

The storage duration can be used for representing the time difference between the recording time of the information recorded by the sending end in the network equipment and the current time; the recording time may be used to represent the time when the sender recorded information is first stored in the network device. For example, the time when the sender id is obtained by first parsing and the sender record information is recorded may be used as the recording time of the sender record information.

In some embodiments, the target time period may be any time period set in advance, for example, 1 day, 7 days, or 1 month.

In some embodiments, the target duration may be calculated by the following equation (12):

wherein,

the target duration is represented, f represents a preset data rejection coefficient, wherein f can be any positive number smaller than 1, dl represents the current quantity of the sending end recording information currently stored by the network equipment, db represents a preset quantity threshold, and t1 represents preset data rejection interval time.

Therefore, the larger the current quantity of the currently stored sending end record information is, the smaller the calculated target duration is, and the more the sending end record information is removed.

In some embodiments, the step of reasonably removing the sending-end record information currently stored by the network device may be triggered and executed based on a target event, or may be executed periodically.

Therefore, the sending end record information currently stored in the network equipment can be reasonably eliminated according to the target duration so as to reduce resource occupation.

Fig. 4 is a block diagram illustrating a log data processing apparatus 1100 according to an exemplary embodiment, and as shown in fig. 4, the apparatus 1100 may include:

a log obtaining module 1101, configured to obtain log data generated by the network device; the log data comprises a log recorded with a sending end identifier of sending end equipment, and the sending end equipment is equipment for sending a message to the network equipment or sending a message flowing through the network equipment;

a log processing module 1102, configured to determine a target log type and the sender identifier corresponding to the log data;

an index analysis module 1103, configured to determine an abnormal analysis index corresponding to the sender identifier according to the target log type;

and an exception handling module 1104, configured to execute an exception handling operation according to the sender identifier when the exception analysis indicator is greater than or equal to a preset indicator threshold.

In some embodiments, the index analysis module 1103 is configured to determine an abnormal analysis adjustment parameter corresponding to the target log type according to a log parameter correspondence; the log parameter corresponding relation comprises a corresponding relation between the target log type and the abnormal analysis adjustment parameter; and determining an abnormal analysis index corresponding to the sending end identifier according to the abnormal analysis adjustment parameter.

In some embodiments, the index analysis module 1103 is configured to, when a historical abnormal analysis index corresponding to the sender identifier is obtained, determine the abnormal analysis index according to the historical abnormal analysis index and the abnormal analysis adjustment parameter, and use the abnormal analysis index as a new historical abnormal analysis index; or, under the condition that the historical abnormal analysis index corresponding to the sending end identifier is not obtained, determining the abnormal analysis index corresponding to the sending end identifier according to the abnormal analysis adjustment parameter, and using the abnormal analysis index as the historical abnormal analysis index corresponding to the sending end identifier.

In some embodiments, the apparatus further comprises:

fig. 5 is a block diagram illustrating another log data processing apparatus 1100 according to an exemplary embodiment, and as shown in fig. 5, the apparatus 1100 may further include:

an initialization module 1105 for determining a plurality of candidate log types; the candidate log type comprises the target log type; acquiring a first data volume generated by each candidate log type in a first time period; the first time period is a historical time period before the current moment; calculating to obtain a first anomaly analysis adjustment parameter corresponding to each candidate log type according to the first data volume; the more the first data volume is, the smaller the first anomaly analysis adjustment parameter obtained by calculation is; and taking the first abnormal analysis adjustment parameter as an abnormal analysis adjustment parameter corresponding to the candidate log type to obtain the corresponding relation of the log parameters.

In some embodiments, as shown in fig. 5, the apparatus 1100 may further include:

a dynamic planning module 1106, configured to obtain the number of logs counted in the second time period and a third data amount in a third time period for each candidate log type; the second time period and the third time period are both time periods before the current moment, and the third time period comprises the second time period; determining a second weight corresponding to the third data volume and a first weight corresponding to the log quantity, wherein the first weight is greater than or equal to the second weight; calculating to obtain a second anomaly analysis adjustment parameter corresponding to each candidate log type according to the third data volume, the second weight, the log number and the first weight; and taking the second abnormal analysis adjustment parameter as an abnormal analysis adjustment parameter corresponding to the candidate log type to obtain the log parameter corresponding relation.

a data removing module 1107, configured to obtain a current amount of sender record information currently stored in the network device; each sending end record message comprises a sending end identifier, a record time and a historical abnormal analysis index corresponding to the sending end identifier, wherein the record time is the time when the sending end record message is stored to the network equipment for the first time; determining a target duration according to the current number and a preset number threshold under the condition that the current number is greater than or equal to the preset number threshold; and deleting the sending end recording information with the storage duration being greater than or equal to the target duration from the network equipment, wherein the storage duration is used for representing the time difference between the recording time and the current time.

In some embodiments, the log processing module 1102 is configured to perform text parsing on the log data according to a preset type parameter, and determine a corresponding target log type in the log data; and according to a preset regular expression, performing text analysis on the log data to obtain the sending end identifier.

With regard to the apparatus in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be described in detail here.

Fig. 6 is a block diagram illustrating an electronic device 2000 in accordance with an example embodiment. As shown in fig. 6, the electronic device 2000 may include: a processor 2001, memory 2002. The electronic device 2000 may also include one or more of a multimedia component 2003, an input/output (I/O) interface 2004, and a communications component 2005.

The processor 2001 is used for controlling the overall operation of the electronic device 2000 to complete all or part of the steps of the log data processing method. The memory 2002 is used to store various types of data to support operation at the electronic device 2000, such as instructions for any application or method operating on the electronic device 2000, and application-related data, such as contact data, messaging, pictures, audio, video, and the like. The Memory 2002 may be implemented by any type of volatile or non-volatile Memory device or combination thereof, such as Static Random Access Memory (SRAM), electrically Erasable Programmable Read-Only Memory (EEPROM), erasable Programmable Read-Only Memory (EPROM), programmable Read-Only Memory (PROM), read-Only Memory (ROM), magnetic Memory, flash Memory, magnetic disk, or optical disk. The multimedia component 2003 may include a screen and an audio component. Wherein the screen may be, for example, a touch screen and the audio component is used for outputting and/or inputting audio signals. For example, the audio component may include a microphone for receiving an external audio signal. The received audio signal may further be stored in the memory 2002 or transmitted through the communication component 2005. The audio assembly also includes at least one speaker for outputting audio signals. The input/output interface 2004 provides an interface between the processor 2001 and other interface modules, which may be a keyboard, mouse, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 2005 is used for wired or wireless communication between the electronic device 2000 and other devices. Wireless Communication, such as Wi-Fi, bluetooth, near Field Communication (NFC), 2G, 3G, 4G, 5G, NB-IOT, eMTC, or other 6G, or a combination of one or more of them, which is not limited herein. The corresponding communication component 2005 may therefore include: wi-Fi module, bluetooth module, NFC module etc..

In an exemplary embodiment, the electronic Device 2000 may be implemented by one or more Application Specific Integrated Circuits (ASICs), digital Signal Processors (DSPs), digital Signal Processing Devices (DSPDs), programmable Logic Devices (PLDs), field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components for performing the log data Processing method.

In another exemplary embodiment, there is also provided a computer-readable storage medium including a computer program or program instructions stored thereon, which when executed by a processor, implement the steps of the log data processing method described above. For example, the computer readable storage medium may be the memory 2002 described above including program instructions that are executable by the processor 2001 of the electronic device 2000 to perform the log data processing method described above. The computer readable storage medium may be, for example, a non-transitory computer readable storage medium such as a ROM, a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

In another exemplary embodiment, a computer program product is also provided, which comprises a computer program executable by a programmable apparatus, the computer program having code portions for performing the above-mentioned log data processing method when executed by the programmable apparatus.

Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method of processing log data, the method comprising:

acquiring log data generated by network equipment; the log data comprises a sending end identifier of sending end equipment, and the sending end equipment is equipment for sending a message to the network equipment or sending a message flowing through the network equipment;

determining an abnormal analysis index corresponding to the sending end identification according to the target log type;

2. The method of claim 1, wherein the determining the anomaly analysis indicator corresponding to the sender id according to the target log type comprises:

3. The method according to claim 2, wherein the determining the anomaly analysis indicator corresponding to the sender id according to the anomaly analysis adjustment parameter includes:

under the condition that a historical abnormal analysis index corresponding to the sending end identifier is obtained, determining the abnormal analysis index according to the historical abnormal analysis index and the abnormal analysis adjustment parameter, and taking the abnormal analysis index as a new historical abnormal analysis index; or,

4. The method of claim 2, wherein the log parameter correspondence is generated by:

5. The method of claim 4, wherein the log parameter correspondence is updated by:

acquiring the number of logs counted in a second time period and a third data volume in a third time period of each candidate log type; the second time period and the third time period are both time periods before the current moment, and the third time period comprises the second time period;

calculating to obtain a second anomaly analysis adjustment parameter corresponding to each candidate log type according to the third data volume, the second weight, the log quantity and the first weight;

6. The method of claim 3, further comprising:

acquiring the current quantity of the sending end record information currently stored by the network equipment; each sending end record message comprises a sending end identifier, a record time and a historical abnormal analysis index corresponding to the sending end identifier, wherein the record time is the time when the sending end record message is stored to the network equipment for the first time;

7. The method according to any one of claims 1 to 6, wherein the determining the target log type and the sender identity corresponding to the log data comprises:

and according to a preset regular expression, performing text analysis on the log data to obtain the sending end identification.

8. An apparatus for processing log data, the apparatus comprising:

9. An electronic device, comprising:

a memory having a computer program stored thereon;

a processor for executing the computer program in the memory to implement the steps of the method of any one of claims 1 to 7.

10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 7.