CN112306812A

CN112306812A - Log processing method and device, computer equipment and storage medium

Info

Publication number: CN112306812A
Application number: CN202011254874.1A
Authority: CN
Inventors: 丁志翔; 付星
Original assignee: Wuhan Yuexuebang Network Technology Co ltd
Current assignee: Wuhan Yuexuebang Network Technology Co ltd
Priority date: 2020-11-11
Filing date: 2020-11-11
Publication date: 2021-02-02

Abstract

The present disclosure provides a log processing method, apparatus, computer device and storage medium, wherein the method comprises: acquiring a log to be processed; acquiring an alarm condition for alarming an abnormal log input by a user based on the generated attribute information of the log to be processed; determining first log attribute information of the log to be processed, which corresponds to the alarm condition; and when the first log attribute information meets the alarm condition, generating alarm information of the log to be processed based on the first log attribute information. According to the method and the device for processing the log to be processed, the log to be processed is subjected to alarm judgment according to the acquired condition for alarming the log to be processed input by the user and the acquired attribute information of the log to be processed, the alarm condition of the abnormal log can be managed and set by the user, the flexibility and the applicability of alarm condition configuration are improved, and the reasonability of an alarm result is improved.

Description

Log processing method and device, computer equipment and storage medium

Technical Field

The present disclosure relates to the field of computer technologies, and in particular, to a log processing method and apparatus, a computer device, and a storage medium.

Background

Logging in an application generally serves two purposes: troubleshot and display program running status. The good log recording mode can not only accurately position the problem, but also ensure the simplicity of the recording mode, so that the application log alarming platform can be used for rapidly helping an administrator to find the problem of application operation and timely solve the problem.

In the existing application log alarm platform, an abnormal log generated in an application running process is collected through a filebeat (log data collector) and sent to an ELK (elastic search, logstack, Kibana, three open source frames) for storage, and then the stored abnormal log record is queried and an alarm is realized by utilizing an elastAllet (alarm frame).

However, when the above method is used to monitor the abnormal log for alarming, configuration files need to be added to the abnormal log according to the alarm rule of each application, and when the number of applications is too large, the difficulty of management and maintenance is increased due to too many added configuration files.

Disclosure of Invention

The embodiment of the disclosure at least provides a log processing method and device, computer equipment and a storage medium.

In a first aspect, an embodiment of the present disclosure provides a log processing method, including:

acquiring a log to be processed;

acquiring an alarm condition for alarming an abnormal log input by a user based on the generated attribute information of the log to be processed;

determining first log attribute information of the log to be processed, which corresponds to the alarm condition;

and when the first log attribute information meets the alarm condition, generating alarm information of the log to be processed based on the first log attribute information.

In one possible embodiment, the method further comprises:

acquiring a shielding condition for shielding an abnormal log input by a user based on the generated attribute information of the log to be processed;

before determining the first log attribute information of the log to be processed, which corresponds to the alarm condition, the method further comprises the following steps:

determining second log attribute information of the log to be processed, which corresponds to the shielding condition;

and when the second log attribute information meets the shielding condition, executing the step of determining the first log attribute information of the log to be processed, which corresponds to the alarm condition.

In one possible embodiment, the method further comprises:

acquiring an alarm mode for alarming an abnormal log input by a user based on the generated attribute information of the log to be processed;

and displaying the alarm information to an alarm object by using the alarm mode.

In a possible implementation manner, the shielding condition includes a preset first application name, a preset shielding keyword, or a preset shielding period; the second log attribute information includes a name of an application that generates the log to be processed, a generation time of the log to be processed, and

when the second log attribute information meets the shielding condition, the step of determining the first log attribute information of the log to be processed corresponding to the alarm condition is executed, and the step of determining the first log attribute information of the log to be processed corresponding to the alarm condition comprises the following steps:

based on the second log attribute information, judging whether the name of the application generating the log to be processed in the second log attribute information is the same as the preset first application name, or judging whether the generation time of the log to be processed in the second log attribute information is in the preset shielding time period, or judging whether the log to be processed comprises the preset shielding keyword;

and if the name of the application generating the log to be processed is the same as the preset first application name, or the generation time of the log to be processed is not in the preset shielding time period, or the log to be processed does not include the preset shielding keyword, executing the step of determining first log attribute information of the log to be processed, which corresponds to the alarm condition.

In a possible implementation manner, the alarm condition includes a preset second application name, a preset alarm keyword, a preset alarm time interval, or a preset abnormal number threshold, and the first log attribute information includes a name of an application that generates the log to be processed, a number of the log to be processed corresponding to the second application name, and

when the first log attribute information meets the alarm condition, generating alarm information for the log to be processed based on the first log attribute information comprises the following steps:

based on the first log attribute information, judging whether the name of the application generating the log to be processed in the first log attribute information is the same as the preset second application name, or judging whether the log to be processed comprises the preset alarm keyword, or judging whether the number of the log to be processed corresponding to the second application name in the first log attribute information meets the preset abnormal number threshold, or judging whether an alarm is generated at the preset alarm time interval according to the name of the application generating the log to be processed;

if the name of the application generating the log to be processed is the same as the preset second application name, or the log to be processed comprises the preset alarm keyword, or the number of the log to be processed corresponding to the second application name in the first log attribute information meets the preset abnormal number threshold, or it is determined that no alarm is generated in the preset alarm time interval according to the name of the application generating the log to be processed; generating alarm information for the log to be processed based on the first log attribute information.

In one possible embodiment, the alarm modes include preset alarm modes corresponding to each alarm level, and

displaying the alarm information to an alarm object by using the alarm mode, wherein the method comprises the following steps:

determining a target alarm level according to the frequency generated by the log to be processed;

and displaying the alarm information to the alarm object by using a preset alarm mode matched with the target alarm level.

In a possible implementation manner, the displaying the alarm information to the alarm object by using a preset alarm manner matched with a target alarm level includes:

determining a corresponding alarm service based on the preset alarm mode;

and displaying the alarm information to the alarm object by utilizing the alarm service.

In a second aspect, an embodiment of the present disclosure further provides a log processing apparatus, including:

the acquisition module is used for acquiring logs to be processed and acquiring alarm conditions for alarming abnormal logs input by a user based on the generated attribute information of the logs to be processed;

the determining module is used for determining first log attribute information of the log to be processed, which corresponds to the alarm condition;

and the generating module is used for generating the alarm information of the log to be processed based on the first log attribute information when the first log attribute information meets the alarm condition.

In a possible embodiment, the apparatus further comprises:

the judging module is used for acquiring a shielding condition for shielding the abnormal log input by a user based on the generated attribute information of the log to be processed;

before the determining module determines the first log attribute information of the log to be processed, which corresponds to the alarm condition, the method further comprises the following steps:

determining second log attribute information of the log to be processed, which corresponds to the shielding condition; and when the second log attribute information meets the shielding condition, executing the step of determining the first log attribute information of the log to be processed, which corresponds to the alarm condition.

In a possible embodiment, the apparatus further comprises:

the display module is used for acquiring an alarm mode for alarming the abnormal log input by a user based on the generated attribute information of the log to be processed; and displaying the alarm information to an alarm object by using the alarm mode.

the judging module is configured to judge, based on the second log attribute information, whether a name of an application that generates the log to be processed in the second log attribute information is the same as the preset first application name, or judge whether the generation time of the log to be processed in the second log attribute information is within the preset shielding time period, or judge whether the log to be processed includes the preset shielding keyword;

the determining module is configured to determine, based on the first log attribute information, whether a name of an application that generates the to-be-processed log in the first log attribute information is the same as a preset second application name, or determine whether the to-be-processed log includes the preset alarm keyword, or determine whether a number of the to-be-processed logs corresponding to the second application name in the first log attribute information satisfies a preset abnormal number threshold, or determine whether an alarm has been generated at the preset alarm time interval according to the name of the application that generates the to-be-processed log;

the display module is used for determining a target alarm level according to the frequency generated by the log to be processed; and displaying the alarm information to the alarm object by using a preset alarm mode matched with the target alarm level.

In a possible implementation manner, the display module is specifically configured to determine a corresponding alarm service based on the preset alarm mode; and displaying the alarm information to the alarm object by utilizing the alarm service.

In a third aspect, this disclosure also provides a computer device, a processor, and a memory, where the memory stores machine-readable instructions executable by the processor, and the processor is configured to execute the machine-readable instructions stored in the memory, and when the machine-readable instructions are executed by the processor, the machine-readable instructions are executed by the processor to perform the steps in the first aspect or any one of the possible implementations of the first aspect.

In a fourth aspect, this disclosure also provides a computer-readable storage medium having a computer program stored thereon, where the computer program is executed to perform the steps in the first aspect or any one of the possible implementation manners of the first aspect.

For the description of the effects of the log processing apparatus, the computer device, and the computer-readable storage medium, reference is made to the description of the log processing method, which is not repeated here.

The log processing method, device, computer equipment and storage medium provided by the embodiment of the disclosure adopt the alarm condition for judging the abnormal log input by the user based on the acquired log to be processed and the generated attribute information thereof, realize the free configuration of the alarm condition by the user by receiving the alarm condition input by the user, improve the flexibility and applicability of the configuration of the alarm condition, further determine the first log attribute information included in the generated attribute information of the log to be processed according to the alarm condition, generate the alarm information when the first log attribute information meets the alarm condition input by the user based on the comparison result of the first log attribute information and the alarm condition, judge the log to be processed by utilizing the alarm condition, improve the rationality and reliability of the alarm, and store the abnormal log generated in the process of acquiring and applying by filebot in the prior art and sending the abnormal log to an ELK, then utilizing Elastalert to inquire the stored abnormal log record and realizing the mode of alarming, aiming at different applications, needing to add configuration files according to the alarming condition, when the application quantity is too large, the added configuration files are too much to improve the difficulty of management and maintenance, obtaining the condition of alarming the abnormal log input by the user according to the obtained log to be processed and the attribute information thereof, realizing the alarming condition of the abnormal log which can be managed and set by the user, improving the flexibility of the configuration of the alarming condition, realizing the autonomous management of the application by the user, improving the applicability of the log monitoring system, further determining the first log attribute information of the corresponding log to be processed according to the obtained alarming condition, utilizing the method of comparing the first log attribute information with the alarming condition input by the user when determining that the first log attribute information meets the alarming condition input by the user, and alarm information is generated, and alarm judgment is performed on the log to be processed according to the set alarm conditions, so that the reasonability and reliability of the alarm are improved.

Further, the log processing method provided by the embodiment of the disclosure may further obtain a shielding condition of the abnormal log input by the user according to the obtained generation attribute information of the log to be processed, determine the second log attribute information according to the shielding condition, judge the log to be processed based on the shielding condition and the second log attribute information, filter the abnormal log which does not need to be alarmed and the abnormal log which the user considers to be unimportant through the shielding condition input by the user, and improve the accuracy and rationality of alarming for the abnormal log.

Furthermore, the log processing method provided by the embodiment of the disclosure may further obtain an alarm mode of an abnormal log input by a user according to the generated attribute information of the obtained log to be processed, determine the alarm mode according to the generated alarm information, improve the diversity of the alarm modes, distinguish the priority level of the log to be processed according to different alarm modes, and improve the rationality of monitoring the abnormal log.

In order to make the aforementioned objects, features and advantages of the present disclosure more comprehensible, preferred embodiments accompanied with figures are described in detail below.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly described below, and the drawings herein incorporated in and forming a part of the specification illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the technical solutions of the present disclosure. It is appreciated that the following drawings depict only certain embodiments of the disclosure and are therefore not to be considered limiting of its scope, for those skilled in the art will be able to derive additional related drawings therefrom without the benefit of the inventive faculty.

Fig. 1 is a schematic view illustrating an application scenario of a log processing method provided by an embodiment of the present disclosure;

FIG. 2 is a flow chart illustrating a log processing method provided by an embodiment of the present disclosure;

fig. 3 is a flowchart illustrating a specific implementation procedure of a log processing method according to an embodiment of the present disclosure;

fig. 4 shows a schematic diagram of a log processing apparatus provided by an embodiment of the present disclosure;

fig. 5 shows a schematic diagram of a computer device provided by an embodiment of the present disclosure.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present disclosure more clear, the technical solutions of the embodiments of the present disclosure will be described clearly and completely with reference to the drawings in the embodiments of the present disclosure, and it is obvious that the described embodiments are only a part of the embodiments of the present disclosure, not all of the embodiments. The components of embodiments of the present disclosure, as generally described and illustrated herein, may be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present disclosure is not intended to limit the scope of the disclosure, as claimed, but is merely representative of selected embodiments of the disclosure. All other embodiments, which can be derived by a person skilled in the art from the embodiments of the disclosure without making creative efforts, shall fall within the protection scope of the disclosure.

Furthermore, the terms "first," "second," and the like in the description and in the claims, and in the drawings described above, in the embodiments of the present disclosure are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that the embodiments described herein may be practiced otherwise than as specifically illustrated or described herein.

Reference herein to "a plurality or a number" means two or more. "and/or" describes the association relationship of the associated objects, meaning that there may be three relationships, e.g., a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship.

According to research, a certain abnormal log monitoring mode needs to be set for an abnormal log generated in the application running process, when the abnormal log generated by the application is monitored, an alarm is given according to the content of the abnormal log, in the prior art, a commonly used abnormal log monitoring mode is that the abnormal log generated in the application running process is collected through filebeat and sent to an ELK for storage, and then the stored abnormal log record is inquired and the alarm is given by utilizing Elastalert, however, in the mode, a configuration file needs to be added to each application according to the alarm rule of the application, and when the application number is too large, the difficulty of management and maintenance is improved due to the fact that the added configuration files are too many.

Based on the above research, the present disclosure provides a log processing method, apparatus, computer device, and storage medium, the condition for alarming the abnormal log input by the user is obtained according to the obtained log to be processed and the attribute information thereof, the alarming condition of the abnormal log can be managed and set by the user is realized, the flexibility of the alarming condition configuration is improved, the autonomous management of the application by the user is realized, the applicability of the log monitoring system is improved, and furthermore, determining the first log attribute information of the corresponding log to be processed according to the acquired alarm condition, comparing the first log attribute information with the alarm condition input by the user, generating alarm information when the first log attribute information is determined to satisfy the alarm condition input by the user, and the alarm judgment is carried out on the log to be processed through the set alarm condition, so that the reasonability and the reliability of the alarm are improved.

The above-mentioned drawbacks are the results of the inventor after practical and careful study, and therefore, the discovery process of the above-mentioned problems and the solutions proposed by the present disclosure to the above-mentioned problems should be the contribution of the inventor in the process of the present disclosure.

It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures.

To facilitate understanding of the present embodiment, first, a detailed description is given to a log processing method disclosed in an embodiment of the present disclosure, where an execution subject of the log processing method provided in the embodiment of the present disclosure is generally a computing device with a certain computing capability, and the computing device includes, for example: a terminal device, which may be a User Equipment (UE), a mobile device, a User terminal, a cellular phone, a cordless phone, a Personal Digital Assistant (PDA), a handheld device, a computing device, a vehicle mounted device, a wearable device, or a server or other processing device. In some possible implementations, the log processing method may be implemented by a processor calling computer readable instructions stored in a memory.

It should be noted that specific terms mentioned in the embodiments of the present disclosure include: filebeat: the log data collector is a local file and can monitor a log directory or a specific log file and forward the log directory or the specific log file to kafka or Logstatsh and the like;

logstatsh: the central data flow engine of the ELK is used for filtering data with different formats collected from different targets and outputting the filtered data to different destinations in a supporting manner;

ES: the Elasticisearch is a search server based on Lucene;

kafka: a high throughput distributed publish-subscribe messaging system that can handle all the activity flow data of a consumer in a website;

UI (user interface): user Interface refers to the overall design of human-computer interaction, operation logic and beautiful Interface of software.

Example one

To facilitate understanding of the present embodiment, an application scenario of the log processing method disclosed in the embodiment of the present disclosure is first introduced, and as shown in fig. 1, an application scenario schematic diagram of the log processing method provided in the embodiment of the present disclosure is shown. A user logs in the application server 12 through an application client installed in the terminal device 11, where the application client may be a browser of a web page, an application client installed in a terminal device, such as a mobile phone, a tablet computer, or the like, or a web page or an applet embedded in an application program. The terminal device 11 and the application server 12 are communicatively connected through a network, which may be a local area network, a cellular network, a wide area network, and the like. The terminal device 11 may be a portable device (e.g., a mobile phone, a tablet, a notebook Computer, etc.) or a Personal Computer (PC), and the application server 12 may be any device capable of providing internet services.

When an abnormal log to be processed is generated in the running process of the application, wherein Filebeat collects each generated abnormal log to be processed and sends the log to a kafka message queue, the kafka message queue is deployed on a service provided by a micro server 13 in a micro-service architecture system, an application server 12 acquires the abnormal log to be processed from the kafka message queue, analyzes the abnormal log to be processed to obtain generation attribute information and log content of the abnormal log to be processed, stores the log content of the log to be processed in an ES (ES), the ES is deployed on the service provided by a micro server 14 in the micro-service architecture system, and then acquires a shielding condition, an alarm condition and an alarm mode input by a user through a UI (user interface) of the application based on the generation attribute information of the log to be processed, wherein the shielding condition, the alarm condition and the alarm mode can be input according to the needs of the user and the actual running situation of the application, the method comprises the steps of inputting and changing at any time through an applied UI interface, determining second log attribute information of a log to be processed according to a shielding condition input by a user, comparing the second attribute information with the shielding condition, determining first attribute information of the log to be processed according to the warning condition input by the user under the condition that the log to be processed is determined not to be shielded, comparing the first attribute information with a warning condition, generating warning information according to the first attribute information and log content and storing the warning information in an ES under the condition that the warning condition is determined to be met, determining a specific warning mode according to the warning mode input by the user and the log content, calling a corresponding warning service according to the determined warning mode to complete the warning of the abnormal log to be processed, wherein the warning service is also provided by a micro server deployed in a micro-service architecture system, and automatically configuring warning rules by the user, the applicability of the log monitoring system is improved, the autonomous management of the user on the application is realized, and the difficulty of application monitoring and alarming is reduced. It should be noted that the micro service architecture includes not only the micro server 13 and the micro server 14, but also other micro servers, and the number of the micro servers specifically included may be determined according to the function provided by the application.

In addition, the micro-service architecture system is a technology for deploying applications and services in the cloud, and provides a set of basic architecture, the architecture enables the micro-services to be deployed, operated and upgraded independently, and the system architecture enables the micro-services and the micro-services to be structurally loosely coupled and functionally represent a unified whole. The micro-service can be realized by a Docker container, one physical machine can simultaneously operate a plurality of containers, or only one container can be operated, and each container bears one micro-service, so that a complex micro-service framework is realized.

Example two

The following describes a log processing method provided by an embodiment of the present disclosure, taking an execution subject as a computer device as an example.

As shown in fig. 2, a flowchart of a log processing method provided in an embodiment of the present disclosure may include the following steps:

s201: and acquiring a log to be processed.

In specific implementation, when an application generates abnormal to-be-processed logs in the running process, a local log collector Filebeat collects each abnormal to-be-processed log generated by the application and sends the abnormal to-be-processed log to a kafka message queue for storage, wherein the purpose of storing the to-be-processed logs in the kafka message queue is to prevent the abnormal to-be-processed logs generated in a certain time period from being excessive and directly sent to an application server, which may cause downtime of the application server and confusion of application functions. Further, the application server deployed in the abnormal log monitoring and alarming system can acquire each abnormal log to be processed stored in the kafka message queue, so that the log to be processed can be processed in the next step.

For example, application a generates an exception pending log, the contents of which are as follows:

2020-10-26

13:31:20.784

ERROR [ GlobalExceptionHandler:25] [ http-nio-28688-exec-30] - [ TID: ] [ TID: N/A ] unknown anomaly! e { }

org.apache.catalina.connector.ClientAbortException:java.io.IOException:Broken pipe

at

org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:333)。

The log to be processed of the exception collected by the filebed may be:

appName:A

message 2020-10-2613:31: 20.784ERROR [ GlobalExceptionHandler:25] [ http-nio-28688-exec-30] - [ TID: ] [ TID: N/A ] unknown anomaly! e { }

at

Ap, cat, connector, export buffer, realWriteBytes (export buffer, java:333), and then Filebeat sends the content to the kafka message queue for storage, from which the application server can obtain the log to be processed.

S202: and acquiring an alarm condition for alarming the abnormal log input by the user based on the generated attribute information of the log to be processed.

In this step, after acquiring the log to be processed, the application server parses the log to obtain the generated attribute information and the abnormal log content of the log to be processed, and stores the abnormal log content in the ES, in a possible implementation manner, the generated attribute information may include an application name, a generation time, and the like of the log to be processed, and further, based on the generated attribute information of the log to be processed, the alarm condition for alarming the abnormal log, which is input by a user through an UI interface of the application, stored in the application server is acquired, in a specific implementation, the alarm condition may include a preset second application name, a preset alarm keyword, a preset alarm time interval, or a preset threshold of abnormal number, it should be noted that the alarm condition for alarming the abnormal log input by the user may include a plurality of sets of associated sub-alarm conditions, each sub-alarm condition may also have a corresponding alarm rule name. In addition, if the alarm condition for alarming the abnormal log input by the user through the UI interface of the application is not obtained based on the generated attribute information of the log to be processed, it indicates that the application abnormality corresponding to the log to be processed may not be processed, and the log to be processed is ignored.

It should be noted that the alarm condition (and the shielding condition mentioned later) for alarming the abnormal log, which is input by the user through the UI interface of the application, may be input after receiving the log to be processed, or may be input in advance and stored in the application. For example, for a certain application frequently used, an alarm condition may be input in advance, and input at each time is not required, and for some applications not frequently used, input may be performed in real time, which is not described in detail.

Taking the abnormal log to be processed mentioned in step S201 as an example, the application name a of the log to be processed obtained by analysis may determine that a second application name a corresponding to the first sub-alarm condition needs to be obtained, the corresponding preset alarm keyword is java. The second application name corresponding to the second sub-alarm condition is a, the corresponding preset alarm time interval is 60S, the preset abnormal quantity threshold is 200, the corresponding alarm rule name may be that an abnormality exists in the application 60S, of course, a third sub-alarm condition may also be included, and the setting quantity of the specific sub-alarm condition may be set by a user according to the specific use condition of each application.

In specific implementation, before acquiring an alarm condition for alarming an abnormal log input by a user, a shielding condition for shielding the abnormal log input by the user through an application UI interface needs to be acquired according to generated attribute information obtained by parsing, where the shielding condition may include a preset first application name, a preset shielding keyword, or a preset shielding time period, further, the application server may determine, according to the content of the abnormal log, second log attribute information corresponding to the acquired shielding condition, and in specific implementation, the second log attribute information may include an application name for generating a log to be processed, and a generation time of the log to be processed. In addition, if the shielding condition for shielding the abnormal log input by the user through the UI interface of the application is not obtained according to the generated attribute information obtained by analysis, the alarm condition for alarming the abnormal log input by the user can be directly obtained.

And further, comparing the determined second log attribute information with the obtained shielding condition, judging whether the second log attribute information meets the shielding condition, and when the second log attribute information does not meet the shielding condition, indicating that the application abnormity corresponding to the log to be processed cannot be shielded, continuing to judge the next step. In specific implementation, based on the second log attribute information, firstly, judging whether the name of the application generating the log to be processed in the second log attribute information is the same as the preset first application name, wherein the judgment is to ensure that the used shielding condition is specific to the application generating the log to be processed, if the shielding condition of other applications is utilized to perform shielding judgment on the log to be processed, the obtained judgment result directly influences the management of a user on the application, under the condition that the name of the application generating the log to be processed is determined to be the same as the preset first application name, judging whether the log to be processed comprises the preset shielding keyword according to the abnormal log content corresponding to the log to be processed, under the condition that the log to be processed does not comprise the preset shielding keyword, judging whether the log to be processed is in the preset shielding time period according to the generation time of the log to be processed in the second log attribute information, and under the condition that the generation time of the log to be processed is determined not to be in the preset shielding time period, executing the next operation on the log to be processed, or else, if any judgment result is opposite to the judgment result, directly shielding the application exception corresponding to the log to be processed. It should be noted that the judgment order regarding each masking condition is not limited to the judgment order referred to in the above-described disclosed embodiment.

Taking the abnormal log to be processed mentioned in step S201 as an example, the application name a of the log to be processed obtained by analysis may determine that the first application name a corresponding to the shielding condition to be obtained, the shielding keyword is java. io. outexception, and the shielding period is 0 point to 6 points, then the application name of the log to be processed corresponding to the second log attribute information is determined to be a according to the shielding condition, the generation time of the log to be processed is 2020-10-2613:31:20, further, the name of the application generating the log to be processed may be determined to be the same as the first application name, it is determined according to the content of the abnormal log that the log to be processed does not include the shielding keyword, and the generation time of the log to be processed is 2020-10-2613:31:20 which does not belong to the shielding period 0 point to 6 points, which indicates that the application abnormality corresponding to the log to be processed may not be shielded, the next operation on the log to be processed is performed.

S203: and determining first log attribute information of the log to be processed, which corresponds to the alarm condition.

In specific implementation, after acquiring an alarm condition for alarming an abnormal log input by a user, the application server may determine, according to the alarm condition and abnormal log content corresponding to the log to be processed, first log attribute information corresponding to the alarm condition, for example, it may be determined according to the alarm condition in step S202 that the corresponding first log attribute information includes a name of an application that generates the log to be processed and a number of logs to be processed corresponding to a second application name.

S204: and when the first log attribute information meets the alarm condition, generating alarm information of the log to be processed based on the first log attribute information.

In the step, after the alarm condition input by the user and the corresponding first log attribute information are obtained, the alarm condition is compared with the first log attribute information, and whether the application exception corresponding to the log to be processed needs to be shielded is judged according to the comparison result.

In an implementation, it is first required to determine whether the name of an application that generates a log to be processed in first log attribute information is the same as a preset second application name, where this determination is to ensure that a used alarm condition is for the application that generates the log to be processed, and if the log to be processed is subjected to shielding determination by using a shielding condition of another application, an obtained determination result directly affects management of the application by a user, and in a case where it is determined that the name of the application that generates the log to be processed is the same as the preset second application name, it is determined whether the log to be processed includes a preset alarm keyword according to abnormal log content corresponding to the log to be processed, and in an implementation, if the user does not set the preset alarm keyword, this determination process may be directly skipped, and a next determination is performed; in another embodiment, if a preset alarm keyword input by a user is obtained, and when the log to be processed does not include the preset alarm keyword, if other sub-alarm conditions input by the user exist, the judgment is continuously performed by using the other sub-alarm conditions, and if the other sub-alarm conditions input by the user do not exist, the application abnormality corresponding to the log to be processed does not need to be alarmed, and the log to be processed can be directly ignored. Judging whether the number of logs to be processed corresponding to the name of the second application in the first log attribute information meets a preset abnormal number threshold or not under the condition that the logs to be processed comprise preset alarm keywords, judging whether an alarm is generated in a preset alarm time interval or not according to the name of the application generating the logs to be processed when the number of the logs to be processed meets the preset abnormal number threshold is determined, acquiring historical alarm information which is stored in an ES and generated correspondingly to the name of the application generating the logs to be processed in specific implementation, judging whether the alarm is generated in the preset alarm time interval or not according to the time generated by each alarm information in the historical alarm information, and generating the alarm information of the logs to be processed according to the first log attribute information under the condition that the alarm is not generated in the preset alarm time interval. In specific implementation, the alarm information generated for the log to be processed may include a name of an alarm application, an alarm time, a name of an alarm rule, an exception detail statistic (an exception constant per hour today), exception content, and the like. Taking the abnormal log to be processed mentioned in step S201 as an example, the generated alarm information may be a name of an alarm application: a. the

And (3) warning time: 2020-10-2613:31:25

Alarm rule name: exception present within application 60s

And (3) abnormal detail statistics: number of abnormalities of approximately 60s (300)3 points (1)4 points (0)5 points (0)6 points (0)7 points (0)8 points (0)9 points (0)10 points (4)11 points (2)12 points (0)13 points (310)

Abnormal contents are as follows: 2020-10-2613:31: 20.784ERROR [ GlobalExceptionHandler:25] [ http-nio-28688-exec-30] - [ TID: ] [ TID: N/A ] unknown anomaly! e { }

at

On the contrary, if any judgment result is opposite to the judgment result, when other sub-alarm conditions input by the user exist, the other sub-alarm conditions can be directly utilized for judgment, and if the judgment result opposite to the judgment result is determined to exist according to the other sub-alarm conditions, the application abnormity corresponding to the log to be processed can be directly shielded, and no alarm information is generated; and when other sub-alarm conditions input by the user do not exist, the application exception corresponding to the log to be processed is directly shielded, and no alarm information is generated. It should be noted that the judgment order regarding each alarm condition is not limited to the judgment order referred to in the above-described disclosed embodiments.

Taking the alarm condition mentioned in step S202 as an example, if the name of the application generating the log to be processed in the determined first log attribute information is a and the number of the logs to be processed corresponding to the second application name is 350, it may be determined that the name of the application generating the log to be processed is the same as the second application name, it may be determined that the preset alarm keyword is java, io, ioexception in the log to be processed according to the abnormal log content corresponding to the log to be processed, it may be further determined that the preset abnormal number threshold is 300 according to the number of the logs to be processed being 350, but it is determined that an alarm has been generated at the preset alarm time interval according to the time generated by each alarm information in the history alarm information, it is described that the log to be processed does not satisfy the first sub-alarm condition, the judgment is continued by using the second sub-condition, and it may be determined that an alarm preset by the user in the second sub-alarm condition according to the time generated by each alarm information in the history alarm information And if the time interval does not generate an alarm, generating alarm information of the log to be processed according to the first log attribute information. Further, after the alarm information of the log to be processed is generated, an alarm mode for alarming the abnormal log input by the user through the UI interface of the application needs to be obtained based on the generated attribute information of the log to be processed, and then the alarm mode for the application abnormality corresponding to the log to be processed is determined according to the alarm mode input by the user and the generated alarm information. In an embodiment, the alarm manner input by the user may include a preset alarm manner corresponding to the alarm level corresponding to each log to be processed, and in specific implementation, the satisfied target alarm level may be determined according to the frequency generated by the log to be processed, and then the generated alarm information is sent to the alarm object needing to be notified by using the preset alarm manner corresponding to the target alarm level. It should be noted that each preset alarm mode may correspond to a different alarm service, and each alarm service may be deployed in the micro-service architecture system, where the alarm service may include an instant messaging tool alarm, a voice alarm, a short message alarm, and the like, and after the preset alarm mode is determined, the corresponding alarm service may be determined, and then the micro-server corresponding to the alarm service is called to implement the alarm on the alarm object. For example, if the user configures three types of common, important, and high-quality alarm levels for the application a, the corresponding preset alarm modes may include a first-level alarm mode, a second-level alarm mode, and a third-level alarm mode, where the alarm service corresponding to the first-level alarm mode may be an instant messaging tool alarm, the alarm service corresponding to the second-level alarm mode may be a short message alarm, and the alarm service corresponding to the third-level alarm mode may be a voice alarm.

Taking the alarm information generated in step S204 as an example, it may be determined according to the statistics of the details of the abnormality that 5 abnormalities are generated per second in the frequency generated by the log to be processed, that the target alarm level corresponding thereto is a high-priority alarm and that the alarm service corresponding thereto is a voice alarm, and that the alarm objects are a character C and a robot D according to the alarm mode input by the user, the voice service is invoked to notify the character C and the robot D of the generated alarm information.

According to the log processing method provided by the embodiment of the disclosure, a user can autonomously configure shielding conditions, alarm conditions and alarm modes for application abnormity through an application UI interface, the flexibility and the applicability of application alarm management are improved, unnecessary application abnormity is effectively screened through multi-stage judgment of the obtained logs to be processed, the effectiveness of application abnormity alarm is improved, in addition, different alarm services can be called through different preset alarm modes, the alarm for application abnormity is realized, the emergency degree of application abnormity can be distinguished according to different alarm services, and the timeliness of abnormity processing is ensured.

EXAMPLE III

As shown in fig. 3, a flowchart of a specific implementation process of a log processing method provided in the embodiment of the present disclosure may include the following steps:

s301: and acquiring an exception log.

In this step, when the application generates an exception log in the running process, filebeat collects and sends the exception log to the kafka message queue for storage, and further, the application server obtains the exception log stored in the kafka message queue.

S302: and analyzing the abnormal log and storing the abnormal log into the ES.

In specific implementation, the acquired abnormal log is analyzed, and the generation attribute information and the abnormal content of the abnormal log can be obtained.

S303: and judging whether the abnormal shielding condition can be acquired or not based on the generated attribute information of the abnormal log.

In specific implementation, if the exception masking condition can be obtained based on the generation attribute information of the exception log, step S304 is executed, and if not, step S306 is executed.

S304: and judging whether the abnormal log can be shielded or not by using the abnormal shielding condition.

In specific implementation, second attribute information of the corresponding abnormal log is determined based on the obtained abnormal shielding condition, wherein the second attribute information is obtained from the abnormal content of the abnormal log, whether the abnormal log can meet the abnormal shielding condition is judged according to the abnormal shielding condition and the second attribute information, if not, step S306 is executed, and if yes, step S305 is executed.

S305: the flow ends.

In this step, when it is determined that the abnormal log can satisfy the shielding condition, it is indicated that the application abnormality corresponding to the abnormal log can be ignored, and the process ends.

S306: and judging whether the abnormal alarm condition can be acquired or not based on the generated attribute information of the abnormal log.

If yes, go to step S308, if no, go to step S307.

S307: the flow ends.

In specific implementation, if the abnormal alarm condition may not be obtained based on the generated attribute information of the abnormal log, which indicates that the user may not configure the abnormal alarm condition about the abnormality, the flow of ignoring the abnormal log is ended.

S308: and judging whether the abnormal log can be shielded or not by using the abnormal alarm condition and the historical abnormal stored in the ES.

In specific implementation, based on the obtained abnormal alarm condition, determining first attribute information of an abnormal log corresponding to the abnormal alarm condition, where the first attribute information is obtained from abnormal content of the abnormal log, and determining whether the abnormal log can mask the abnormal log according to the abnormal alarm condition, the first attribute information and historical abnormality stored in the ES, if so, executing step S309, and if not, executing step S310.

S309: the flow ends.

And when the abnormal log is judged not to meet the abnormal alarm condition, the application abnormality corresponding to the abnormal log can be ignored, and the process is ended.

S310: and generating alarm information based on the abnormal content of the abnormal log.

In this step, when it is determined that the abnormal log can satisfy the abnormal alarm condition, alarm information is generated based on the abnormal content of the abnormal log, and the alarm information is stored in the ES.

S311: and acquiring an abnormal alarm mode based on the generated attribute information of the abnormal log.

After the alarm information is generated, the abnormal alarm mode input by the user can be obtained based on the generated attribute information of the abnormal log.

S312: and determining a target abnormal alarm mode according to the alarm information.

In specific implementation, the target alarm level met by the abnormal log can be determined according to the alarm information, and the target abnormal alarm mode of the alarm can be determined according to the target alarm level.

S313: and determining alarm service according to the target abnormal alarm mode.

In this step, since each abnormal alarm mode has a corresponding alarm service, the corresponding alarm service needs to be determined according to the target abnormal alarm mode.

S314: and displaying the alarm information to the alarm object by using the alarm service.

In specific implementation, after the alarm service is determined, the alarm service is used to display the alarm information to the alarm object, and further, the alarm object can process the application generating the abnormality according to the received alarm information.

According to the log processing method provided by the embodiment of the disclosure, by receiving the alarm condition input by the user, the alarm condition can be freely configured by the user, the flexibility and the applicability of the configuration of the alarm condition are improved, the alarm service is called according to the determined target abnormal alarm mode to display the alarm information for the alarm object, and the diversification and the rationality of the alarm are improved.

It will be understood by those skilled in the art that in the method of the present invention, the order of writing the steps does not imply a strict order of execution and any limitations on the implementation, and the specific order of execution of the steps should be determined by their function and possible inherent logic.

Based on the same inventive concept, the embodiment of the present disclosure further provides a log processing apparatus corresponding to the log processing method, and as the principle of the apparatus in the embodiment of the present disclosure for solving the problem is similar to the log processing method in the embodiment of the present disclosure, the implementation of the apparatus may refer to the implementation of the method, and repeated details are not repeated.

As shown in fig. 4, a schematic diagram of a log processing apparatus provided for an embodiment of the present disclosure includes:

an obtaining module 401, configured to obtain a log to be processed and obtain an alarm condition for giving an alarm to an abnormal log, where the alarm condition is input by a user, based on generated attribute information of the log to be processed;

a determining module 402, configured to determine first log attribute information of the log to be processed, where the first log attribute information corresponds to the alarm condition;

a generating module 403, configured to generate alarm information for the to-be-processed log based on the first log attribute information when the first log attribute information satisfies the alarm condition.

In a possible embodiment, the apparatus further comprises:

a determining module 404, configured to obtain a shielding condition for shielding an abnormal log, which is input by a user, based on the generated attribute information of the log to be processed;

before the determining module 402 determines the first log attribute information of the log to be processed, which corresponds to the alarm condition, the method further includes:

In a possible embodiment, the apparatus further comprises:

a display module 405, configured to obtain an alarm manner for alarming an abnormal log input by a user based on the generated attribute information of the log to be processed; and displaying the alarm information to an alarm object by using the alarm mode.

the determining module 404 is configured to determine, based on the second log attribute information, whether a name of an application that generates the to-be-processed log in the second log attribute information is the same as the preset first application name, or determine whether the generation time of the to-be-processed log in the second log attribute information is in the preset shielding time period, or determine whether the to-be-processed log includes the preset shielding keyword;

the determining module 402 is configured to determine, based on the first log attribute information, whether a name of an application that generates the to-be-processed log in the first log attribute information is the same as the preset second application name, or determine whether the to-be-processed log includes the preset alarm keyword, or determine whether the number of the to-be-processed logs corresponding to the second application name in the first log attribute information satisfies the preset abnormal number threshold, or determine whether an alarm has been generated at the preset alarm time interval according to the name of the application that generates the to-be-processed log;

the display module 405 is configured to determine a target alarm level according to a frequency generated by the log to be processed; and displaying the alarm information to the alarm object by using a preset alarm mode matched with the target alarm level.

In a possible implementation manner, the display module 405 is specifically configured to determine a corresponding alarm service based on the preset alarm manner; and displaying the alarm information to the alarm object by utilizing the alarm service.

The description of the processing flow of each module in the device and the interaction flow between the modules may refer to the related description in the above method embodiments, and will not be described in detail here.

An embodiment of the present disclosure further provides a computer device, as shown in fig. 5, which is a schematic structural diagram of the computer device provided in the embodiment of the present disclosure, and includes:

a processor 51 and a memory 52; the memory 52 stores machine-readable instructions executable by the processor 51, the processor 51 being configured to execute the machine-readable instructions stored in the memory 52, the processor 51 performing the following steps when the machine-readable instructions are executed by the processor 51: step S201: acquiring a log to be processed; step S202: acquiring an alarm condition for alarming an abnormal log input by a user based on the generated attribute information of the log to be processed; step S203: determining first log attribute information of the log to be processed corresponding to the alarm condition, and step S204: and when the first log attribute information meets the alarm condition, generating alarm information of the log to be processed based on the first log attribute information.

The storage 52 includes a memory 521 and an external storage 522; the memory 521 is also referred to as an internal memory, and temporarily stores operation data in the processor 51 and data exchanged with an external memory 522 such as a hard disk, and the processor 51 exchanges data with the external memory 522 through the memory 521.

In a possible implementation, the instructions executed by the processor 51 further include:

In a possible embodiment, the processor 51 executes an instruction in which the masking condition includes a preset first application name, a preset masking keyword, or a preset masking period; the second log attribute information includes a name of an application that generates the log to be processed, a generation time of the log to be processed, and

In a possible implementation manner, in the instruction executed by the processor 51, the alarm condition includes a preset second application name, a preset alarm keyword, a preset alarm time interval, or a preset abnormal number threshold, the first log attribute information includes a name of an application that generates the log to be processed, a number of the logs to be processed corresponding to the second application name, and

In one possible embodiment, the instructions executed by the processor 51 include preset alert modes corresponding to each alert level, and

In a possible embodiment, the instructions executed by the processor 51 for presenting the alarm information to the alarm object by using a preset alarm mode matching with the target alarm level include:

determining a corresponding alarm service based on the preset alarm mode;

The specific execution process of the instruction may refer to the steps of the log processing method described in the embodiments of the present disclosure, and details are not described here.

The embodiments of the present disclosure also provide a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the computer program performs the steps of the log processing method described in the above method embodiments. The storage medium may be a volatile or non-volatile computer-readable storage medium.

The computer program product of the log processing method provided in the embodiments of the present disclosure includes a computer-readable storage medium storing a program code, where instructions included in the program code may be used to execute the steps of the log processing method described in the above method embodiments, which may be referred to specifically for the above method embodiments, and are not described herein again. The computer program product may be embodied in hardware, software or a combination thereof. In an alternative embodiment, the computer program product is embodied in a computer storage medium, and in another alternative embodiment, the computer program product is embodied in a Software product, such as a Software Development Kit (SDK), or the like.

It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the system and the apparatus described above may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, apparatus, and method may be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one logical division, and there may be other divisions when actually implemented, and for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of devices or units through some communication interfaces, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present disclosure may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.

The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a non-volatile computer-readable storage medium executable by a processor. Based on such understanding, the technical solution of the present disclosure may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present disclosure. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

Finally, it should be noted that: the above-mentioned embodiments are merely specific embodiments of the present disclosure, which are used for illustrating the technical solutions of the present disclosure and not for limiting the same, and the scope of the present disclosure is not limited thereto, and although the present disclosure is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: any person skilled in the art can modify or easily conceive of the technical solutions described in the foregoing embodiments or equivalent technical features thereof within the technical scope of the present disclosure; such modifications, changes or substitutions do not depart from the spirit and scope of the embodiments of the present disclosure, and should be construed as being included therein. Therefore, the protection scope of the present disclosure shall be subject to the protection scope of the claims.

Claims

1. A log processing method, comprising:

acquiring a log to be processed;

2. The log processing method according to claim 1, further comprising:

3. The log processing method according to claim 1, further comprising:

4. The log processing method according to claim 2, wherein the mask condition includes a preset first application name, a preset mask keyword, or a preset mask period; the second log attribute information includes a name of an application that generates the log to be processed, a generation time of the log to be processed, and

5. The log processing method according to claim 1, wherein the alarm condition includes a preset second application name, a preset alarm key, a preset alarm time interval, or a preset abnormal number threshold, the first log attribute information includes a name of an application that generates the log to be processed, a number of the logs to be processed corresponding to the second application name, and

6. The log processing method according to claim 3, wherein the alarm modes include preset alarm modes corresponding to each alarm level, and

7. The log processing method according to claim 6, wherein the displaying the alarm information to the alarm object by using a preset alarm mode matching with a target alarm level comprises:

determining a corresponding alarm service based on the preset alarm mode;

8. A log processing apparatus, comprising:

9. The apparatus of claim 8, further comprising:

10. The apparatus of claim 8, further comprising:

the display module is used for acquiring an alarm mode for alarming the abnormal log input by a user based on the generated attribute information of the log to be processed;

11. A computer device, comprising: a processor, a memory storing machine readable instructions executable by the processor, the processor for executing the machine readable instructions stored in the memory, the processor performing the steps of the log processing method as claimed in any one of claims 1 to 7 when the machine readable instructions are executed by the processor.

12. A computer-readable storage medium, characterized in that a computer program is stored thereon, which, when being executed by a computer device, performs the steps of the log processing method according to any one of claims 1 to 7.