CN113094250B - Log early warning method and device, electronic equipment and storage medium - Google Patents
Log early warning method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN113094250B CN113094250B CN202110519804.2A CN202110519804A CN113094250B CN 113094250 B CN113094250 B CN 113094250B CN 202110519804 A CN202110519804 A CN 202110519804A CN 113094250 B CN113094250 B CN 113094250B
- Authority
- CN
- China
- Prior art keywords
- log
- data
- early warning
- file
- distributed
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/451—Execution arrangements for user interfaces
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Human Computer Interaction (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application provides a log early warning method, a log early warning device, electronic equipment and a storage medium, and relates to the technical field of computers. The method comprises the following steps: acquiring log data; performing data processing on the log data based on early warning rules through a source flow starting processing framework to obtain statistical data; and calling the domain-specific language rule computing capability through the open source stream processing framework to perform early warning computation on the statistical data so as to obtain an early warning computation result, wherein the early warning computation result is used for indicating whether early warning is required for the log data or not. According to the method, log data is read in real time by using the open source stream processing framework, processing statistics on the log data is carried out according to the early warning rule, and the self-defined calculation capability of a set of formulas realized by combining DSL is combined, so that the automatic calculation of the early warning rule is realized, hard coding processing of the log by a user is not needed, the operation steps of log early warning configuration are simplified, and the efficiency of log analysis is greatly improved.
Description
Technical Field
The application relates to the technical field of computers, in particular to a log early warning method, a log early warning device, electronic equipment and a storage medium.
Background
Various applications are constantly generating logs, and problems can be quickly located based on the logs, and the problems can also be quickly analyzed based on the logs, such as generating a readability report based on log data to help make decisions. In these scenes, the real-time monitoring and early warning of the log is a function which many application systems must have, so that the log can be found out in time and the damage can be stopped in time when the system has problems. The existing early warning of the log is non-automatic or semi-automatic, the early warning configuration and calculation cannot be carried out based on the original information of the log end to end, and the problems of complex early warning configuration steps and low early warning configuration efficiency exist.
Disclosure of Invention
Accordingly, an object of the embodiments of the present application is to provide a log early warning method, apparatus, electronic device, and storage medium, so as to solve the problems in the prior art that the log early warning configuration step is complex and the early warning configuration efficiency is low, where the early warning configuration and calculation cannot be performed based on the log original information end to end.
The embodiment of the application provides a log early warning method, which comprises the following steps: acquiring log data; performing data processing on the log data based on early warning rules through a source flow starting processing framework to obtain statistical data; and calling the domain-specific language rule computing capability through the open source stream processing framework to perform early warning computation on the statistical data so as to obtain an early warning computation result, wherein the early warning computation result is used for indicating whether early warning is required for the log data or not.
In the implementation mode, the open source stream processing framework is utilized to read the log data in real time, processing statistics on the log data is carried out according to the early warning rule, and the self-defined calculation capability of a set of formulas realized by combining DSL is realized, so that the automatic calculation of the early warning rule is realized, hard coding processing on the log by a user is not needed, the operation steps of log early warning configuration are simplified, and the efficiency of log analysis is greatly improved.
Optionally, the acquiring log data includes: the log data is read from a distributed log system.
In the implementation manner, the log data is read through the distributed log system, so that high throughput of the log data can be ensured, and the log data can be stably and durably stored.
Optionally, before the reading of the log data from the distributed log system, the method further comprises: matching the log file through a regular expression; monitoring a file directory of the log file through a kernel to determine a newly added log file under the file directory; and writing the newly added log file into the distributed log system as the log data.
In the implementation manner, the newly added log can be accurately obtained through matching the log files and screening the newly added log files, so that the efficiency of log obtaining and subsequent log analysis is improved.
Optionally, the writing the newly added log file as the log data into the distributed log system includes: determining a point location file of the newly added log file, wherein the point location file is used for recording the file name and the acquisition position of the newly added log file; writing the point location file into a locally stored offset.bak file; determining, by fdatasync, that the point location file has been written to the local store; renaming offset. Bak to offset by rename system call; and writing the log data into a corresponding message set in the distributed log system for storage.
In the implementation manner, the information of files to be collected and where the file content is collected is recorded in real time based on the point location files when the log is collected, so that the high availability of the point location files and the collected log files is ensured.
Optionally, the reading the log data from the distributed log system includes: creating a stream execution environment; configuring cluster information of the distributed log system; configuring a data reading application program interface of the open source stream processing framework based on a message set of the distributed log system; the distributed log system is added to the stream execution environment as a source of consumption data to cause the data reading application program interface to read the log data from the distributed log system.
In the above implementation manner, the stream processing setting of the distributed log system and the open source stream processing framework is performed before the log data is read, so that the log data can be automatically and efficiently read from the distributed log system.
Optionally, before the data processing of the log data based on the early warning rule by the open source stream processing framework, the method further comprises: reading the information set of the distributed log system through an information set information reading application program interface; based on the information of the message set, reading the log content in the distributed log system through a log content reading application program interface; and displaying the log content on the front-end display page through a text browsing box of the front-end JavaScript.
In the implementation manner, through displaying the log content, a user can browse the log content in real time in the front-end window, so that the efficiency and the visibility of log analysis are improved.
Optionally, before the data processing of the log data based on the early warning rule by the open source stream processing framework, the method further comprises: displaying a field of the log content in a log browsing frame of the display page so that a user determines a specified field from the log content through the log browsing frame; displaying operators in an operator browsing frame of the display page so that the user determines a specified operator from the operators through the operator browsing frame; and displaying the specified field and the specified operator in a formula editing frame of the display page so that the user configures the early warning rule through the formula editing frame.
In the implementation mode, through displaying the log browsing frame, the operator browsing frame and the formula editing frame, a user does not need to carry out hard coding processing on the log by the user, does not need to write an early warning rule, only needs to select log information in the browser, can complete automatic early warning of the log through configuring the early warning rule by visual information, and simplifies configuration steps of log early warning.
The embodiment of the application also provides a log early warning device, which comprises: the log acquisition module is used for acquiring log data; the data processing module is used for carrying out data processing on the log data based on the early warning rule through a source flow starting processing framework so as to obtain statistical data; and the early warning calculation module is used for carrying out early warning calculation on the statistical data through the open source stream processing framework calling domain-specific language rule calculation capability so as to obtain an early warning calculation result, wherein the early warning calculation result is used for indicating whether the log data needs to be early warned or not.
In the implementation mode, the open source stream processing framework is utilized to read the log data in real time, processing statistics on the log data is carried out according to the early warning rule, and the self-defined calculation capability of a set of formulas realized by combining DSL is realized, so that the automatic calculation of the early warning rule is realized, hard coding processing on the log by a user is not needed, the operation steps of log early warning configuration are simplified, and the efficiency of log analysis is greatly improved.
Optionally, the log obtaining module is specifically configured to: the log data is read from a distributed log system.
In the implementation manner, the log data is read through the distributed log system, so that high throughput of the log data can be ensured, and the log data can be stably and durably stored.
Optionally, the log early warning device further includes: the log acquisition module is used for matching log files through regular expressions; monitoring a file directory of the log file through a kernel to determine a newly added log file under the file directory; and writing the newly added log file into the distributed log system as the log data.
In the implementation manner, the newly added log can be accurately obtained through matching the log files and screening the newly added log files, so that the efficiency of log obtaining and subsequent log analysis is improved.
Optionally, the log collection module is specifically configured to: determining a point location file of the newly added log file, wherein the point location file is used for recording the file name and the acquisition position of the newly added log file; writing the point location file into a locally stored offset.bak file; determining, by fdatasync, that the point location file has been written to the local store; renaming offset. Bak to offset by rename system call; and writing the log data into a corresponding message set in the distributed log system for storage.
In the implementation manner, the information of files to be collected and where the file content is collected is recorded in real time based on the point location files when the log is collected, so that the high availability of the point location files and the collected log files is ensured.
Optionally, the log obtaining module is specifically configured to: creating a stream execution environment; configuring cluster information of the distributed log system; configuring a data reading application program interface of the open source stream processing framework based on a message set of the distributed log system; the distributed log system is added to the stream execution environment as a source of consumption data to cause the data reading application program interface to read the log data from the distributed log system.
In the above implementation manner, the stream processing setting of the distributed log system and the open source stream processing framework is performed before the log data is read, so that the log data can be automatically and efficiently read from the distributed log system.
Optionally, the log early warning device further includes: the display module is used for reading the information set information of the distributed log system through the information set information reading application program interface; based on the information of the message set, reading the log content in the distributed log system through a log content reading application program interface; and displaying the log content on the front-end display page through a text browsing box of the front-end JavaScript.
In the implementation manner, through displaying the log content, a user can browse the log content in real time in the front-end window, so that the efficiency and the visibility of log analysis are improved.
Optionally, the display module is further configured to: displaying a field of the log content in a log browsing frame of the display page so that a user determines a specified field from the log content through the log browsing frame; displaying operators in an operator browsing frame of the display page so that the user determines a specified operator from the operators through the operator browsing frame; and displaying the specified field and the specified operator in a formula editing frame of the display page so that the user configures the early warning rule through the formula editing frame.
In the implementation mode, through displaying the log browsing frame, the operator browsing frame and the formula editing frame, a user does not need to carry out hard coding processing on the log by the user, does not need to write an early warning rule, only needs to select log information in the browser, can complete automatic early warning of the log through configuring the early warning rule by visual information, and simplifies configuration steps of log early warning.
The embodiment of the application also provides electronic equipment, which comprises a memory and a processor, wherein the memory stores program instructions, and the processor executes the steps in any implementation mode when reading and running the program instructions.
Embodiments of the present application also provide a readable storage medium having stored therein computer program instructions which, when read and executed by a processor, perform the steps of any of the above implementations.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments of the present application will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and should not be considered as limiting the scope, and other related drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a schematic flow chart of a log file collection step according to an embodiment of the present application.
Fig. 2 is a flowchart illustrating a log file writing step according to an embodiment of the present application.
Fig. 3 is a schematic diagram of a display diagram of a front-end page according to an embodiment of the present application.
Fig. 4 is a flow chart of a log early warning method according to an embodiment of the present application.
Fig. 5 is a schematic block diagram of a log early warning device according to an embodiment of the present application.
Icon: 20-a log early warning device; 21-a log acquisition module; 22-a data processing module; and 23-an early warning calculation module.
Detailed Description
The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application.
The embodiment of the application provides a log early warning method, which can acquire log data and perform log early warning on the log data.
Referring to fig. 1, fig. 1 is a schematic flow chart of a log file collecting step provided in an embodiment of the present application, where the log file collecting step may specifically be as follows:
step S111: the log files are matched by regular expressions.
It should be understood that the log has a certain format, the fields in the log data are all in text format, and the log collection is a program for collecting the data from the source end to the destination end, and in this embodiment, the program of the destination end may be a distributed log system.
The simplest way of detecting and finding the log files is that the user directly lists the files to be collected in the configuration files, but the log files are usually dynamically generated, the mode of only relying on fixed file catalogues and file formats is low in efficiency and large in workload, and the log files have certain naming rules, so that the log files can be collected in a regular expression mode.
For example, where the name of the log file is xai.log, xai.log-20121-06-17, or similar such forms, regular expressions may be used to match such file instances, for example: the xai.log (- [0-9] {4} - [0-9] {2} - [0-9] {2 }) has regular expression screening to determine which files are log files that need to be collected and which files are other types of files that do not need to be collected.
Step S112, the file directory of the log file is monitored through the kernel to determine the newly added log file under the file directory.
After the log files are found through regular expression detection, it is further required to determine which of the log files are newly added in real time in a current dynamic mode, and in this embodiment, a Linux kernel can be utilized to provide a high-efficiency Inotify mechanism, and a kernel is utilized to monitor changes of the files under a file directory of the log files so as to determine the newly added log files.
Step S113: and writing the newly added log file into the distributed log system as log data.
When the log file is collected and written into the distributed log system, the collected log file and the collected position of each time need to be recorded in real time, and in the embodiment, the file name and the corresponding collection position of the log file can be recorded through the point location file.
In order to ensure reliable writing of the point location file, in this embodiment, update recording is performed on the point location file, so that the newly added log file is written into the distributed log system, where specific steps may be as shown in fig. 2, fig. 2 is a schematic flow chart of a log file writing step provided in an embodiment of the present application, and the steps may specifically be as follows:
step S1131: and determining a point location file of the newly added log file, wherein the point location file is used for recording the file name and the acquisition position of the newly added log file.
Step S1132: and writing the point location file into a locally stored offset.
Step S1133: it is determined by fdatasync that the point location file has been written to the local storage.
Step S1134: the offset. Bak is renamed to be offset by a rename system call.
Step S1135: and writing the log data into a corresponding message set in the distributed log system for storage.
The distributed log system in this embodiment may be Kafka, which is a high throughput distributed publish-subscribe message system, and may process all action stream data of a consumer in a website, specifically, it is a distributed, multi-partition, multi-copy supported, and Zookeeper-based distributed message stream platform, which is also a open source publish-subscribe mode based message engine system.
So far, the log information in the Kafka can be read and checked, in order to better check the log information, the pre-warning configuration is performed based on the log information, and the real-time browsing can be performed on the message aggregate message of the Kafka, namely the Topic information, and the specific steps can be as follows:
step S114: and reading the information set of the distributed log system through an information set information reading application program interface.
Specifically, the message set information, that is, topic information, is acquired by getadmincient (). Descaletopics.
Step S115: based on the information of the message set, the log content in the distributed log system is read through a log content reading application program interface.
Specifically, log contents of different partitions in the distributed log system are obtained through get (topicName). Parts ().
Step S116: and displaying the log content on the front-end display page through a text browsing box of the front-end JavaScript.
Alternatively, in this embodiment, a text browsing box textArea may be used to display the log, and a sliding box may be used to browse more log contents.
It should be understood that the steps S111-S113 and the steps S114-S116 are not in a sequential relationship, but in a parallel relationship, and the execution sequence of the two steps is not limited.
Since the early warning rules are needed to be used in the subsequent log early warning calculation, the early warning rules of the log content are needed to be configured in advance, a certain log field value can be selected, early warning is performed on a certain field value or early warning is performed after certain statistics is performed on the field value.
Optionally, a field of the log content is displayed in a log view box of the display page, so that the user determines a specified field from the log content through the log view box.
The specified field determined by the user through field selection can be a keyword, a field or a piece of text.
Optionally, in this embodiment, in order to enable the user to perform visual configuration on the log early warning rule, an operator is displayed in an operator browsing frame of the display page, and a specified field and a specified operator are displayed in a formula editing frame of the display page, so that the user configures the early warning rule through the formula editing frame.
For example, early warning is performed on the appearance of one field, after the appointed field is selected, an early warning rule is configured on the appointed field through an operator browsing frame and a formula editing frame, taking a tagError field as an example, if the alarm occurs for 5 times, the tagError is selected firstly, then the operator larger than the right is selected, and then 5 is input into the editing frame, and the configuration is finished, namely, the tagError is >5. If the fields need to be pre-warned after being gathered according to a certain time, the tagError field can be selected, the time granularity is selected, the operator larger than the operator in the operator browsing frame is selected, the operator is edited 5 in the editing area of the formula editing frame, and the configuration is finished, namely tagerror.1 min >5.
Wherein, the operator browsing frame can support numerical operations such as addition, subtraction, multiplication, division, average value, absolute value, maximum value, minimum value operation and the like; support logical operations, such as greater than, less than, greater than or equal to, less than or equal to, operations, and the like, and AND, OR, NOT, etc.; text operations are supported, such as operations that are null, not null, contain, not contain, start with …, not start with …, end with …, not end with …, regular, and the like.
The text browsing frame, the operator browsing frame and the formula editing frame in the front-end display page may be shown in fig. 3, and fig. 3 is a schematic diagram of a display diagram of the front-end page according to an embodiment of the present application.
After the acquisition of log data and the configuration of the early warning rule are completed, log early warning calculation based on the early warning rule can be realized through the cooperation of the open source stream processing frame and the distributed log system, please refer to fig. 4, fig. 4 is a flow diagram of a log early warning method provided by the embodiment of the application, and specific steps of the log early warning method can be as follows:
step S12: log data is obtained.
Optionally, in this embodiment, the open source stream processing framework used for log acquisition and subsequent log early warning calculation may be a link, and the core of the open source stream processing framework is a distributed stream data stream engine written in Java and Scala. The Flink executes any stream data program in a data parallel and pipeline manner, and the pipeline runtime system of the Flink can execute batch processing and stream processing programs.
Specifically, this step S12 may include the following sub-steps:
step S121: a stream execution environment is created.
Specifically, the link may perform data processing based on a single field, or may process a single field according to a time window, where the log is collected in real time, and if the log is to be pre-warned in real time based on rules, a stream execution environment needs to be created, and the stream execution environment is acquired by using streamexecutionenvironment () and a checkpoint enable checkpoint is set.
Step S122: and configuring cluster information of the distributed log system.
Specifically, taking the example of the distributed log system as Kafka, the specific code of the configuration cluster information can be as follows:
Properties props＝new Properties()；
props.setProperty("bootstrap.servers","hadoopa1:9092,hadoopa2:9092")；
props.setProperty("zookeeper.connect","hadoopa1:2181,hadoopa2:2181")；
props.setProperty("group.id","kafka_to_hdfs")。
step S123: the data reading application interface of the open stream processing framework is configured based on the message set of the distributed log system.
Specifically, the above-described data reading application program interface may be that of flinkkafkaConsumer010< String > con= new FlinkKafkaConsumer010< > ("log 1", new SimpleStringSchema (), tips), where log1 represents Topic, and set to read only the latest data con.
Step S124: the distributed log system is added to the stream execution environment as a source of consumer data to cause the data reading application program interface to read log data from the distributed log system.
Specifically, examples of code that a consumer data source adds to a stream execution environment are: dataStream < String > stream = env.
Step S14: and carrying out data processing on the log data based on the early warning rule through a source flow starting processing framework so as to obtain statistical data.
According to the conditions of the configured early warning rules, data processing is performed based on an API (Application Programming Interface, application program interface) provided by the Flink, for example, the configured early warning rules are tagError.1 min >5, which indicates that the fields need to be counted according to minutes, then the timeWindow of the Flink is used for accumulating 1-minute window data, and the Window function is used for acquiring accumulated data and acquiring statistical data.
Step S16: and calling the special language rule computing capability of the field through the source flow processing framework to perform early warning computation on the statistical data so as to obtain an early warning computation result, wherein the early warning computation result is used for indicating whether early warning is required for log data or not.
For example, if the configured early warning rule is a single field, if tagError >5, the data processing is performed by using the flatMap, and then the computing capability of the domain-specific language rule is called to perform early warning computation.
Where domain specific language (Domain Specified Language, DSL) refers to a computer language that focuses on a certain application domain, unlike the common cross-domain general purpose computer language, domain specific language is used only in certain specific domains, such as HTML for displaying web pages, and Emac LISP language used by Emacs.
Specifically, the present embodiment implements a set of DSL rule calculation APIs using a scale parsing composition sub:
(1) First defining the grammar of rule calculation and the priority of operators, such as multiplication budget is larger than addition and subtraction operation, the grammar shows the order of expression calculation.
(2) Programming the grammar to realize. Such as: there may be N multiple and or relationships between expressions and expressions, the code being represented as follows:
(3) Calling the paramal in the scalea package scalea. Uteil. Paraming. Combiner to compile the expression, and writing an expression calculation interface, wherein the packaged function form is as follows: public boolean getBool (Map < String, double > params, string formats).
(4) And according to field data read by the Flink or data after statistical calculation, the field data or the data after statistical calculation are transmitted into the Map, and an interface is called to return a calculation result of log early warning.
In order to cooperate with the log early warning method provided by the embodiment of the present application, the embodiment of the present application further provides a log early warning device 20.
Referring to fig. 5, fig. 5 is a schematic block diagram of a log early warning device according to an embodiment of the present application.
The log early warning device 20 includes:
a log obtaining module 21, configured to obtain log data;
a data processing module 22, configured to perform data processing on the log data based on the early warning rule by using the source flow processing framework to obtain statistical data;
the early warning calculation module 23 is configured to perform early warning calculation on the statistical data by calling a domain-specific language rule calculation capability through the open source stream processing framework, so as to obtain an early warning calculation result, where the early warning calculation result is used to indicate whether early warning is required for log data.
Alternatively, the log obtaining module 21 is specifically configured to: log data is read from the distributed log system.
Optionally, the log early warning device 20 further includes: the log acquisition module is used for matching log files through regular expressions; monitoring a file directory of the log file through the kernel to determine a newly added log file under the file directory; and writing the newly added log file into the distributed log system as log data.
Optionally, the log collection module is specifically configured to: determining a point location file of the newly added log file, wherein the point location file is used for recording the file name and the acquisition position of the newly added log file; writing the point location file into a locally stored offset. Determining, by the fdatasync, that the point location file has been written to the local store; renaming offset. Bak to offset by rename system call; and writing the log data into a corresponding message set in the distributed log system for storage.
Optionally, the log obtaining module is specifically configured to: creating a stream execution environment; configuring cluster information of a distributed log system; configuring a data reading application program interface of a source flow processing framework based on a message set of a distributed log system; the distributed log system is added to the stream execution environment as a source of consumer data to cause the data reading application program interface to read log data from the distributed log system.
Optionally, the log early warning device further includes: the display module is used for reading the information set information of the distributed log system through the information set information reading application program interface; based on the information of the message set, reading the log content in the distributed log system through a log content reading application program interface; and displaying the log content on the front-end display page through a text browsing box of the front-end JavaScript.
Optionally, the display module is further configured to: displaying a field of the log content in a log browsing frame of the display page, so that a user determines a specified field from the log content through the log browsing frame; displaying operators in an operator browsing frame of the display page so that a user determines a specified operator from the operators through the operator browsing frame; and displaying the specified fields and the specified operators in a formula editing box of the display page so that a user configures the early warning rules through the formula editing box.
The embodiment of the application also provides electronic equipment, which comprises a memory and a processor, wherein the memory stores program instructions, and when the processor reads and runs the program instructions, the processor executes the steps in any one of the log early warning methods provided by the embodiment.
It should be understood that the electronic device may be a personal computer (Personal Computer, PC), tablet computer, smart phone, personal digital assistant (Personal Digital Assistant, PDA), or the like, having a logic computing function.
The embodiment of the application also provides a readable storage medium, wherein the readable storage medium stores computer program instructions, and the computer program instructions execute the steps in the log early warning method when being read and run by a processor.
In summary, the embodiment of the application provides a log early warning method, a log early warning device, an electronic device and a storage medium, wherein the method comprises the following steps: acquiring log data; performing data processing on the log data based on early warning rules through a source flow starting processing framework to obtain statistical data; and calling the domain-specific language rule computing capability through the open source stream processing framework to perform early warning computation on the statistical data so as to obtain an early warning computation result, wherein the early warning computation result is used for indicating whether early warning is required for the log data or not.
According to the method and the device, through automatic acquisition of logs, the logs are stored in a distributed log system such as a Kafka cluster, a JS is used for developing a log browsing function at the front end, visual configuration of early warning rules is carried out on pages, log data are read in real time by using a source flow processing framework such as a Flink, certain processing statistics is carried out according to early warning rule information, and a set of formula custom computing capacity is realized by combining DSL, so that visual configuration and automatic computation of the early warning rules are realized. The user does not need to carry out hard coding processing on the log, does not need to write an early warning rule, and can complete automatic early warning of the log by only automatically selecting log information in a browser and configuring the early warning rule. The scheme greatly improves timeliness of problem discovery during log processing, and early warning can be immediately configured for calculation based on log output, so that complex development steps are not needed, and efficiency of log analysis is greatly improved.
In the several embodiments provided in the present application, it should be understood that the disclosed apparatus may be implemented in other manners. The apparatus embodiments described above are merely illustrative, for example, block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of devices according to various embodiments of the present application. In this regard, each block in the block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams, and combinations of blocks in the block diagrams, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
In addition, functional modules in the embodiments of the present application may be integrated together to form a single part, or each module may exist alone, or two or more modules may be integrated to form a single part.
The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. The present embodiment therefore also provides a readable storage medium having stored therein computer program instructions which, when read and executed by a processor, perform the steps of any one of the methods of block data storage. Based on this understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a RanDom Access Memory (RAM), a magnetic disk or an optical disk, or other various media capable of storing program codes.
The above description is only an example of the present application and is not intended to limit the scope of the present application, and various modifications and variations will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application. It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further definition or explanation thereof is necessary in the following figures.
It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.
Claims (4)
1. A log early warning method, the method comprising:
acquiring log data;
performing data processing on the log data based on early warning rules through a source flow starting processing framework to obtain statistical data;
the statistics data are subjected to early warning calculation through the open source stream processing framework calling domain-specific language rule calculation capability so as to obtain early warning calculation results, wherein the early warning calculation results are used for indicating whether early warning is required for the log data or not;
the acquiring log data includes:
reading the log data from a distributed log system;
before the reading of the log data from the distributed log system, the method further comprises:
matching the log file through a regular expression;
monitoring a file directory of the log file through a kernel to determine a newly added log file under the file directory;
writing the newly added log file into the distributed log system as the log data;
the writing the newly added log file as the log data into the distributed log system includes:
determining a point location file of the newly added log file, wherein the point location file is used for recording the file name and the acquisition position of the newly added log file;
writing the point location file into a locally stored offset.bak file;
determining, by fdatasync, that the point location file has been written to the local store;
renaming offset. Bak to offset by rename system call;
writing the log data into a corresponding message set in the distributed log system for storage;
the reading the log data from the distributed log system includes:
creating a stream execution environment;
configuring cluster information of the distributed log system;
configuring a data reading application program interface of the open source stream processing framework based on a message set of the distributed log system;
adding the distributed log system as a source of consumption data to the stream execution environment to cause the data reading application program interface to read the log data from the distributed log system;
before the data processing of the log data based on the pre-warning rules by the open source processing framework, the method further comprises:
reading the information set of the distributed log system through an information set information reading application program interface;
based on the information of the message set, reading the log content in the distributed log system through a log content reading application program interface;
displaying the log content on a front-end display page through a text browsing frame of front-end JavaScript;
before the data processing of the log data based on the pre-warning rules by the open source processing framework, the method further comprises:
displaying a field of the log content in a log browsing frame of the display page so that a user determines a specified field from the log content through the log browsing frame;
displaying operators in an operator browsing frame of the display page so that the user determines a specified operator from the operators through the operator browsing frame;
and displaying the specified field and the specified operator in a formula editing frame of the display page so that the user configures the early warning rule through the formula editing frame.
2. A log alert device, the device comprising:
the log acquisition module is used for acquiring log data; the acquiring log data includes:
reading the log data from a distributed log system;
before the reading of the log data from the distributed log system, further comprising:
matching the log file through a regular expression;
monitoring a file directory of the log file through a kernel to determine a newly added log file under the file directory;
writing the newly added log file into the distributed log system as the log data;
the writing the newly added log file as the log data into the distributed log system includes:
determining a point location file of the newly added log file, wherein the point location file is used for recording the file name and the acquisition position of the newly added log file;
writing the point location file into a locally stored offset.bak file;
determining, by fdatasync, that the point location file has been written to the local store;
renaming offset. Bak to offset by rename system call;
writing the log data into a corresponding message set in the distributed log system for storage;
the reading the log data from the distributed log system includes:
creating a stream execution environment;
configuring cluster information of the distributed log system;
configuring a data reading application program interface of a source flow processing framework based on a message set of the distributed log system;
adding the distributed log system as a source of consumption data to the stream execution environment to cause the data reading application program interface to read the log data from the distributed log system;
the data processing module is used for carrying out data processing on the log data based on the early warning rule through a source flow starting processing framework so as to obtain statistical data; before the log data is processed by the open source processing framework based on the early warning rule, the method further comprises:
reading the information set of the distributed log system through an information set information reading application program interface;
based on the information of the message set, reading the log content in the distributed log system through a log content reading application program interface;
displaying the log content on a front-end display page through a text browsing frame of front-end JavaScript;
before the log data is processed by the open source processing framework based on the early warning rule, the method further comprises:
displaying a field of the log content in a log browsing frame of the display page so that a user determines a specified field from the log content through the log browsing frame;
displaying operators in an operator browsing frame of the display page so that the user determines a specified operator from the operators through the operator browsing frame;
displaying the specified fields and the specified operators in a formula editing box of the display page so that the user configures the early warning rules through the formula editing box
And the early warning calculation module is used for carrying out early warning calculation on the statistical data through the open source stream processing framework calling domain-specific language rule calculation capability so as to obtain an early warning calculation result, wherein the early warning calculation result is used for indicating whether the log data needs to be early warned or not.
3. An electronic device comprising a memory and a processor, the memory having stored therein program instructions which, when executed by the processor, perform the steps of the method of claim 1.
4. A storage medium having stored therein computer program instructions which, when executed by a processor, perform the steps of the method of claim 1.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110519804.2A CN113094250B (en) | 2021-05-12 | 2021-05-12 | Log early warning method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110519804.2A CN113094250B (en) | 2021-05-12 | 2021-05-12 | Log early warning method and device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113094250A CN113094250A (en) | 2021-07-09 |
| CN113094250B true CN113094250B (en) | 2023-08-18 |
Family
ID=76665548
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110519804.2A Active CN113094250B (en) | 2021-05-12 | 2021-05-12 | Log early warning method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113094250B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113901093A (en) * | 2021-08-25 | 2022-01-07 | 北京思特奇信息技术股份有限公司 | Service call log relation analysis method and system based on memory cache |
| CN114205215B (en) * | 2021-12-06 | 2024-07-05 | 湖北天融信网络安全技术有限公司 | Data pre-analysis method and device |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104618328A (en) * | 2014-12-29 | 2015-05-13 | 厦门欣欣信息有限公司 | Network security protection method and device |
| CN109408347A (en) * | 2018-09-28 | 2019-03-01 | 北京九章云极科技有限公司 | A kind of index real-time analyzer and index real-time computing technique |
| CN110245158A (en) * | 2019-06-10 | 2019-09-17 | 上海理想信息产业（集团）有限公司 | A kind of multi-source heterogeneous generating date system and method based on Flink stream calculation technology |
| CN111858278A (en) * | 2020-07-08 | 2020-10-30 | 北京国联视讯信息技术股份有限公司 | Log analysis method and system based on big data processing and readable storage device |
| CN112434949A (en) * | 2020-11-25 | 2021-03-02 | 平安普惠企业管理有限公司 | Service early warning processing method, device, equipment and medium based on artificial intelligence |
-
2021
- 2021-05-12 CN CN202110519804.2A patent/CN113094250B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104618328A (en) * | 2014-12-29 | 2015-05-13 | 厦门欣欣信息有限公司 | Network security protection method and device |
| CN109408347A (en) * | 2018-09-28 | 2019-03-01 | 北京九章云极科技有限公司 | A kind of index real-time analyzer and index real-time computing technique |
| CN110245158A (en) * | 2019-06-10 | 2019-09-17 | 上海理想信息产业（集团）有限公司 | A kind of multi-source heterogeneous generating date system and method based on Flink stream calculation technology |
| CN111858278A (en) * | 2020-07-08 | 2020-10-30 | 北京国联视讯信息技术股份有限公司 | Log analysis method and system based on big data processing and readable storage device |
| CN112434949A (en) * | 2020-11-25 | 2021-03-02 | 平安普惠企业管理有限公司 | Service early warning processing method, device, equipment and medium based on artificial intelligence |
Non-Patent Citations (1)
| Title |
|---|
| 基于工业业务的ICS高交互蜜罐技术研究与威胁情报分析;赵春辉;《中国优秀硕士学位论文全文数据库 信息科技辑》(第8期);第I138-65页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113094250A (en) | 2021-07-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8468391B2 (en) | Utilizing log event ontology to deliver user role specific solutions for problem determination | |
| US20210042170A9 (en) | Automatic registration of empty pointers | |
| US8140573B2 (en) | Exporting and importing business objects based on metadata | |
| US11550628B2 (en) | Performing runbook operations for an application based on a runbook definition | |
| CN113094250B (en) | Log early warning method and device, electronic equipment and storage medium | |
| US8671110B1 (en) | Collaborative modeling environment | |
| US20200183681A1 (en) | Method for a software development system | |
| US10146749B2 (en) | Tracking JavaScript actions | |
| CN110647322B (en) | List rendering method and device, electronic equipment and computer readable medium | |
| CN114528269A (en) | Method, electronic device and computer program product for processing data | |
| US11694092B2 (en) | Reward-based recommendations of actions using machine-learning on telemetry data | |
| US11455461B2 (en) | Self-executing document revision | |
| US20120310893A1 (en) | Systems and methods for manipulating and archiving web content | |
| CN110928941B (en) | Data fragment extraction method and device | |
| US9754033B2 (en) | Optimizing web crawling through web page pruning | |
| CN112906373A (en) | Alarm calculation method and device, electronic equipment and storage medium | |
| CN112597105A (en) | Processing method of file associated object, server side equipment and storage medium | |
| CN109582347B (en) | Method and device for acquiring front-end codes | |
| CN113051333B (en) | Data processing method and device, electronic equipment and storage medium | |
| CN115659045A (en) | User operation identification method and device, storage medium and electronic equipment | |
| US11119761B2 (en) | Identifying implicit dependencies between code artifacts | |
| CN113869882A (en) | Data processing method, device and medium | |
| WO2013039800A1 (en) | Simulation of static members and parameterized constructors on an interface-based api | |
| CN113326004B (en) | Efficient log centralization method and device in cloud computing environment | |
| US10489272B2 (en) | Automatic instrumentation of code |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |