TW202301122A - Log analysis system and log analysis method - Google Patents

Abstract

The present application provides a log analysis system and a log analysis method. The log analysis method includes: obtaining log data; parsing the log data and obtaining log data with failed parsing and log data with successful parsing; performing a data structuring processing on the log data with successful parsing to obtain structured data; correcting the log data with failed parsing and the structured data to obtain revised data; outputting the revised data to a terminal; monitoring a configuration file of the log analysis system according to predetermined time and updating the configuration file of the log analysis system according to monitored change of the configuration file; and realizing a configuration visualization of the log analysis process task by using a web page. This application can assist the log parsing, reduce the difficulty of the log parsing.

Description

Log parsing system and log parsing method

The invention relates to the technical field of log data processing, in particular to a log analysis system and a log analysis method.

With the rapid implementation of 5G technology and the rapid development of the Internet of Things, the field of log data technology needs to collect log data in various ways under the technical background of the Internet of Everything for big data processing and analysis. In the process of collecting log data, due to the various collection media and the different protocols used, the collected digital information is messy and out of order. Uploading such information directly to the cloud will lead to cloud data processing. The difficulty has been greatly increased.

In view of the above, it is necessary to provide a log parsing system and a log parsing method, which can reduce the coupling degree between various modules of the system, facilitate the expansion of the system, and reduce the difficulty of the log parsing process.

The log analysis system includes a plurality of modules, and the plurality of modules include: a log access module for obtaining log data from different communication protocols; a data analysis module for processing the obtained log data Parallel parsing to obtain the log data of failed parsing and the log data of successful parsing; the data parsing module is also used to perform data structural processing on the log data of successful parsing to obtain structured data; the data processing module, It is used to correct the log data that fails to be parsed and the structured data to obtain the corrected data; the data output module is used to output the corrected data to the terminal.

Optionally, the log parsing system includes a plug-in program, which is used to expand the multiple modules; the log parsing system also includes a remote control module, and the remote control module is used to follow the Monitor the setting file of the log analysis system at a predetermined time, and when an updated setting file is monitored, load the updated setting file to update the configuration of the multiple modules; the remote control module The group is also used to control the plurality of modules; the log parsing system further includes a visualization module, and the visualization module realizes configuration visualization of the log parsing system by using a web page.

Optionally, the log access module includes a universal line reader; the remote control module controls the log access module to obtain the logs from different communication protocols based on the universal line reader data, and uniformly access the log data from different communication protocols to the log analysis system.

Optionally, the data parsing module includes a general line interpreter; the remote control module controls the data parsing module, based on the general line interpreter for the log data from different communication protocols Parallel analysis is performed, and based on the general line interpreter, the data structure processing is performed on the successfully parsed log data.

Optionally, the data processing module includes a general line interpreter; the remote control module controls the data processing module, based on the general line interpreter for the log source data and The modification of the structured data includes: deleting or replacing the log data that fails to be parsed, and adding tags to the structured data.

Optionally, the terminal includes a local storage area, a cloud and a console connected through a communication protocol.

Optionally, the remote control module includes a message receiving module, an information parser and a control library.

Optionally, the message receiving module is used to obtain control information for the multiple modules; the information parser is used to analyze the control information to obtain a result of analysis, the result of which includes access control modifier; the control library is used to control the multiple modules according to the access control modifier.

Optionally, the visualization module configures the multiple modules through a web page, and generates setting files of the multiple modules, so that the multiple modules configure the processing the log data; and displaying the running status of the multiple modules on the web page.

The log parsing method utilizes the log parsing system, and the method includes: obtaining log data from different communication protocols; performing parallel parsing on the log data from different communication protocols, obtaining log data of failed parsing and parsing successfully Log data of the log data; carry out data structure processing to the log data of the successful parsing to obtain the structured data; modify the log data of the failed parsing and the structured data to obtain the corrected data; The final data is output to the terminal; the setting file of the log analysis system is monitored according to the predetermined time, and the log analysis system is updated and configured according to the change of the monitored setting file; the above-mentioned log analysis process is realized by using the World Wide Web page Configuration visualization of tasks.

Compared with the conventional technology, in the log analysis system and log analysis method of the present application, the log analysis system forms a plurality of independent modules through decoupling, each module defines a common interface, and the log analysis system uses The process of input, parsing, processing, and output is connected in series with these common interfaces to realize the process in the log analysis method; through the inheritance of the common interface, the function of adapting to various complex application scenarios is realized; through the extraction of interface parameters, the log is parsed The process defines the respective behaviors in the setting document according to the module, and forms a timed task; through the definition of the setting document, the process of the log analysis method is designed into the visual page to reduce the difficulty of use for users. The log parsing system has the characteristics of high cohesion, low coupling, modularization, and scalability, which is conducive to repeated computing upgrades of the system.

In order to more clearly understand the above objects, features and advantages of the present application, the present application will be described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the case of no conflict, the embodiments of the present application and the features in the embodiments can be combined with each other.

A lot of specific details are set forth in the following description to facilitate a full understanding of the application, and the described embodiments are only a part of the embodiments of the application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field to which this application belongs. The terms used herein in the specification of the application are only for the purpose of describing specific embodiments, and are not intended to limit the application.

Referring to FIG. 1 , it is a functional module diagram of the log parsing system 1 provided by the embodiment of the present application. The log analysis system 1 includes a plurality of modules, and the plurality of modules include: a log access module 10, a data analysis module 11, a data processing module 12, and a data output module 13; the log analysis system 1 also includes a remote control module 14 and a visualization module 15, the remote control module 14 is used as the control center of the log analysis system 1. In this embodiment, the log analysis system 1 can be applied to a computer device (such as the computer device 3 shown in FIG. 2 ), and for a computer device that needs to perform log analysis tasks, the computer device of this application can be directly integrated on the computer device. The log parsing function provided by the method may be run on a computer device in the form of a software development kit (Software Development Kit, SDK).

In one embodiment, the log parsing system 1 includes a plug-in program (Plug-in), and the plug-in program can be added or deleted in the form of a patch package during the running of the log parsing task, so as to expand the multiple modules.

In one embodiment, the log access module 10 is used to acquire log data from different communication protocols, and the log data from different communication protocols include log data in different formats and out of order, and the log data It can be accessed by the log access module 10 in the form of being contained in a log file. The log access module 10 includes a general line reader (General Line Reader), which can read the log data in the log file by line; the remote control module 14 uses Controlling the log access module 10 to obtain the log data from different communication protocols through the inheritance of the universal row reader, and uniformly access the log data from different communication protocols to all The log parsing system 1 described above is stored in a predefined database. These different communication protocols may include IP protocol, IMCP protocol, TCP protocol and so on.

In one embodiment, the data parsing module 11 is configured to use parallel technology (Parallel Technology) to perform parallel parsing on the log data obtained in the database, and obtain log data that fail to parse (for example, according to the computer device 3 The pre-defined rule in is judged as the log data of sensitive data thereby parsing failure) and the log data of parsing success; The data parsing module 11 includes a general line interpreter, and the general line interpreter refers to the command line Interpreter (Command Line Interpreter); the remote control module 14 is used to control the data analysis module 11 to parallelize the log data from different communication protocols by inheriting the universal line interpreter Parsing, and based on the universal line interpreter, according to the format predefined in the computer device 3, performing the data structure processing on the successfully parsed log data, and disorganizing the log data in different formats Format it into ordered structured data (Structured Data), and tokenize the log data (Tokenization).

In one embodiment, the data processing module 12 is used to correct the log data that failed to be parsed and the structured data to obtain the corrected data; the data processing module 12 includes general line interpretation The remote control module 14 is used to control the data processing module 12 to correct the log source data that failed to parse and the structured data by inheriting the universal row interpreter. The modification of the log source data of the analysis failure and the structured data includes: retrieving the log source data of the analysis failure, deleting or replacing the retrieved log data of the analysis failure, for example, according to the computer device 3 Pre-defined rules delete or replace keywords in sensitive data. The modifying the log source data and the structured data that failed to parse further includes: retrieving the structured data from the database, according to the attributes and definition rules of the pre-defined data tags in the computer device 3 , to tag the retrieved structured data. The correcting the log source data that fails to be parsed and the structured data further includes: updating the database according to the corrected log data.

In one embodiment, the data output module 13 includes a general data export tool for exporting the corrected data from the database to a terminal, and the terminal can be one of the log analysis system 1 External storage devices, including but not limited to the local storage area of the computer device 3, the cloud connected to the log analysis system 1 through a communication protocol (for example, the MQTT protocol), the console (for example, under Windows Output the operation result of the log parsing system 1 in the CMD console).

In one embodiment, the remote control module 14 is configured to monitor the configuration file of the log analysis system 1 at a predetermined time (for example, every 30 seconds), and when an updated configuration file is detected, , loading the updated setting file to update and configure multiple modules of the log analysis system 1; the remote control module 14 is also used to control the multiple modules. The multiple modules include, but are not limited to, the log access module 10 , data analysis module 11 , data processing module 12 , and data output module 13 .

In one embodiment, the remote control module 14 includes a message receiving module 151 , an information parser 152 and a control library 153 . The message receiving module 151 is used to obtain control information on the multiple modules from different communication protocols, such as communicating with an external message server of the log analysis system 1 (for example, installed on the computer through the MQTT protocol) The message server in the device 3) performs information interaction to obtain the control information; the information parser 152 is used to analyze the control information to obtain an analysis result, and the analysis result includes an access control modifier; The control library 153 is used to control the multiple modules according to the access control modifier, including: controlling the running state of the log analysis system 1 according to the command for updating the configuration file indicated by the access control modifier (eg restart, suspend, start, etc.) or update the log analysis system 1 by downloading the latest version of the software program of the log analysis system 1 .

In other embodiments, the log analysis system 1 may also include a cloud, and the cloud may be installed in the computer device 3 in the form of software. The remote control module 14 sends the current operating state of the log analysis system 1 to the cloud, and the cloud is used to analyze the current operating state of the log analysis system 1 to obtain an analysis result of the operating state The analysis result of the running state is transmitted to the message receiving module 151 through the message server, so as to realize indirect monitoring. The cloud is also used to obtain the control information of the plurality of modules according to the analysis result of the running state, and input the control information into the message receiving module 151 . The message receiving module 151 records the action of receiving the control information into a local log of the log parsing system 1 (for example, a local log file predefined in the database). The information parser 152 is used to match the control behavior in the control information with the interface provided by the control library 153, load control parameters, execute the control process, and record the behavior results generated during the whole process to the described in the local log. The control library 153 provides a common core control execution library, and realizes direct or indirect operation control of the log parsing system 1 through an extended library. The direct control includes a version update library, and the updated setting file is downloaded by calling the interface of the version update library, and the update process of the log analysis system 1 is rerun according to the updated setting file. The indirect control includes controlling the running status of the data parsing tasks of the log parsing system 1. By rewriting the running setting file of each task, the log parsing system 1 periodically (for example, every hour) scans Run the directory where the setting file is located (for example, the directory where the setting file is predefined in the database), and perform operations such as stopping, deleting, restarting or creating a new task in the log analysis system 1 according to the setting file.

In one embodiment, the visualization module 15 realizes the configuration visualization of the log parsing system 1 by using a web page. The visualization module 15 displays the web page through the display of the computer device 3, configures the multiple modules, and generates the setting files of the multiple modules, so that the multiple modules are configured according to The setting file processes the log data; and displays the running status of the multiple modules on the web page. The web page includes a task parameter configuration page, which is used to match each log document to the corresponding log analysis task of the log analysis system 1. After entering a certain log analysis task, the task of the certain log analysis task can be The parameter configuration page configures the log access module 10, data analysis module 11, data processing module 12, and data output module 13 under the log analysis task in order, and saves the setting file after the configuration is completed. What is displayed on the web page is the running status of the log analysis task of the log analysis system 1. For example, in an industrial environment, by installing the log analysis system 1 on the computer device 3 where the production data is located, open the Create a new log analysis task on the World Wide Web page, enter the task parameter configuration page, fill in the forms on the log access parameter configuration page, data analysis configuration page, data processing configuration page, and data output configuration page in turn. Store and exit the task parameter configuration page; select the configured log parsing task, change its running status (such as running, pausing or restarting), and the World Wide Web page can display the status of each log parsing task in the form of a list, For example, the number of log data that was parsed successfully, the number of log data that failed to parse, etc.

In one embodiment, the log parsing system 1 includes a log record library, a log record buried point is set in the log record parse system 1, and a label corresponding to the log record is marked at the log record buried point. The log records generated by the log parsing system 1 during operation will be stored in log files in a local predefined database of the log parsing system 1 . The logs of the log analysis system 1 and the logs of the log analysis process tasks are stored in respective directories. At the same time, the log analysis system 1 realizes the reporting of the operation heartbeat of the log analysis system 1 by defining the log analysis process of the local log file, and the heartbeat information includes the resource usage of the log analysis system 1, and each log analysis task running status, the version of the log parsing system 1, etc.

In one embodiment, the remote control module 14 is an optional module, and the remote control module 14 is configured in the configuration file of the log analysis system 1 , which is separated from the configuration of the log analysis task. The feedback of the execution status of the log analysis system 1 is to filter the log documents of the log analysis system 1 by tags through the log analysis process, and process the records and output them to the cloud to complete the indirect feedback of control.

In one embodiment, the log analysis system 1 can be compiled into an executable file of the log collection terminal platform through cross-compilation in the form of software written in Golang language.

As shown in FIG. 3 , the log parsing method specifically includes the following steps. According to different requirements, the order of the steps in the flow chart can be changed, and some steps can be omitted.

In step S1, the log access module 10 obtains log data from different communication protocols.

In one embodiment, the log data from different communication protocols includes log data in different formats and out of order, and the log data can be received by the log access module 10 in the form of being contained in a log file. enter. The log access module 10 includes a universal line reader, which can read the log data in the log file line by line. The remote control module 14 controls the log access module 10 to obtain the log data from different communication protocols through the inheritance of the universal line reader, and uniformly access the log data from different communication protocols to the log parsing system 1 and stored in a pre-defined database. These different communication protocols may include IP protocol, IMCP protocol, TCP protocol and so on.

In step S2, the data analysis module 11 performs parallel analysis on the log data from different communication protocols, and obtains the log data that fails to be parsed and the log data that succeeds in parsing.

In one embodiment, the data parsing module 11 uses parallel technology to perform parallel parsing on the log data acquired in the database, and obtain log data that fails to parse (for example, according to the predefined rules in the computer device 3 The log data that is judged as sensitive data thereby parsing fails) and the log data that parses successfully; The data parsing module 11 includes a general-purpose line interpreter, and the general-purpose line interpreter refers to a command line interpreter; The remote control module 14 controls the data analysis module 11 to analyze the log data from different communication protocols in parallel by inheriting the universal line interpreter, and based on the general line interpreter, according to The pre-defined format in the computer device 3 performs the data structuring process on the successfully parsed log data, formats the log data with different formats and disorderly into orderly structured data, and processes the log data Tokenization.

In step S3, the data processing module 12 performs data structural processing on the log data that has been successfully parsed to obtain structured data; corrects the log data that fails to be parsed and the structured data to obtain corrected data.

In one embodiment, the data processing module 12 includes a general line interpreter; the remote control module 14 controls the data processing module 12 to implement the The log source data that fails to be parsed and the structured data are corrected. The modification of the log source data of the analysis failure and the structured data includes: retrieving the log source data of the analysis failure, deleting or replacing the retrieved log data of the analysis failure, for example, according to the computer device 3 Pre-defined rules delete or replace keywords in sensitive data. The modifying the log source data and the structured data that failed to parse further includes: retrieving the structured data from the database, according to the attributes and definition rules of the pre-defined data tags in the computer device 3 , to tag the retrieved structured data. The correcting the log source data that fails to be parsed and the structured data further includes: updating the database according to the corrected log data.

In step S4, the data output module 13 outputs the corrected data to the terminal.

In one embodiment, the data output module 13 includes a general data export tool, and the data output module 13 outputs the corrected data from the database to a terminal, and the terminal may be the Storage devices other than the log analysis system 1, including but not limited to the local storage area of the computer device 3, the cloud and the console (such as , output the running result of the log parsing system 1 in the CMD console under Windows).

Step S5 , the remote control module 14 monitors the setting file of the log analysis system according to a predetermined time, and updates the configuration of the log analysis system according to the changes of the monitored setting file.

In one embodiment, the remote control module 14 monitors the configuration file of the log parsing system 1 at a predetermined time (for example, every 30 seconds), and uploads the Import the updated setting file to update and configure multiple modules of the log analysis system 1; the remote control module 14 is also used to control the multiple modules. The multiple modules include, but are not limited to, the log access module 10 , data analysis module 11 , data processing module 12 , and data output module 13 .

In one embodiment, the remote control module 14 includes a message receiving module 151, an information parser 152 and a control library 153, and the message receiving module 151 obtains information from different communication protocols for the multiple modules. control information, such as through the MQTT communication protocol and the message server outside the log analysis system 1 (for example, the message server installed in the computer device 3) for information interaction, to obtain the control information; the information The parser 152 parses the control information to obtain a parsing result, the parsing result includes an access control modifier; the control library 153 controls the multiple modules according to the access control modifier, including : According to the command for updating the configuration file indicated by the access control modifier, control the running state of the log analysis system 1 (such as restart, suspend, start, etc.) or update the software program by downloading the latest version of the log analysis system 1 The log parsing system 1.

In other embodiments, the log analysis system 1 may also include a cloud, and the cloud may be installed in the computer device 3 in the form of software. The remote control module 14 sends the current operating state of the log analysis system 1 to the cloud, and the cloud analyzes the current operating state of the log analysis system 1 to obtain an analysis result of the operating state. The message server transmits the analysis result of the running state to the message receiving module 151 to realize indirect monitoring. The cloud is also used to obtain the control information of the plurality of modules according to the analysis result of the running state, and input the control information into the message receiving module 151 . The message receiving module 151 records the action of receiving the control information into a local log of the log parsing system 1 (for example, a local log file predefined in the database). The information parser 152 corresponds the control behavior in the control information with the interface provided by the control library 153, loads control parameters, executes the control process, and records the behavior results generated during the whole process to the local in the log. The control library 153 provides a common core control execution library, and realizes direct or indirect operation control of the log parsing system 1 through an extended library. The direct control includes a version update library, and the updated setting file is downloaded by calling the interface of the version update library, and the update process of the log analysis system 1 is rerun according to the updated setting file. The indirect control includes controlling the running status of the data parsing tasks of the log parsing system 1. By rewriting the running setting file of each task, the log parsing system 1 periodically (for example, every hour) scans Run the directory where the setting file is located (for example, the directory where the setting file is predefined in the database), and perform operations such as stopping, deleting, restarting or creating a new task in the log analysis system 1 according to the setting file.

In step S6, the visualization module 15 realizes the configuration visualization of the above log parsing process task by using the web page.

In one embodiment, the visualization module 15 configures the multiple modules through a web page, and generates the setting files of the multiple modules, so that the multiple modules can be configured according to the setting files. processing the log data; and displaying the running status of the multiple modules on the web page. The web page includes a task parameter configuration page, which matches the corresponding log parsing task of the log parsing system 1 for each log file. After entering a certain log parsing task, the task configuration page of the certain log parsing task can be The log access module 10, data analysis module 11, data processing module 12, and data output module 13 under the task are sequentially configured, and the configuration file is stored after configuration is completed. What is displayed on the web page is the running status of the log analysis task of the log analysis system 1. For example, in an industrial environment, by installing the log analysis system 1 on the computer device 3 where the production data is located, open the Create a new log analysis task on the World Wide Web page, enter the task parameter configuration page, fill in the forms on the log access parameter configuration page, data analysis configuration page, data processing configuration page, and data output configuration page in turn. Store and exit the task parameter configuration page; select the configured log parsing task, change its running status (such as running, pausing or restarting), and the World Wide Web page can display the status of each log parsing task in the form of a list, For example, the number of log data that was parsed successfully, the number of log data that failed to parse, etc.

The above-mentioned FIG. 3 introduces the log parsing method of the present application in detail. The hardware device architecture for implementing the log parsing method is introduced below in conjunction with FIG. 2 .

It should be understood that the embodiments are only for illustration, and are not limited by the structure in terms of the scope of the patent application.

Referring to FIG. 2 , it is a schematic structural diagram of a computer device provided by an embodiment of the present application. In a preferred embodiment of the present application, the computer device 3 includes a storage 31 and at least one processor 32 . Those skilled in the art should understand that the structure of the computer device shown in Figure 2 does not constitute a limitation of the embodiment of the present application, it can be a bus-type structure or a star structure, and the computer device 3 can also include a ratio More or less other hardware or software, or different arrangements of components are illustrated.

In some embodiments, the computer device 3 includes a terminal capable of automatically performing numerical calculations and/or information processing according to preset or stored instructions, and its hardware includes but not limited to microprocessors, dedicated integrated circuits, Programmable gate arrays, digital signal processors and embedded devices, etc.

It should be noted that the computer device 3 is only an example, and other existing or future electronic products that can be adapted to this application should also be included in the scope of protection of this application and included here by reference.

In some embodiments, the storage 31 is used to store program codes and various data. For example, the storage 31 can be used to store the log analysis system 1 installed in the computer device 3 , and realize high-speed and automatic program or data access during the operation of the computer device 3 . The storage 31 includes a read-only memory (Read-Only Memory, ROM), a programmable read-only memory (Programmable Read-Only Memory, PROM), an erasable programmable read-only memory (Erasable Programmable Read -Only Memory, EPROM), One-time Programmable Read-Only Memory (OTPROM), Electrically-Erasable Programmable Read-Only Memory (EEPROM) , CD-ROM (Compact Disc Read-Only Memory, CD-ROM) or other optical disk storage, disk storage, tape storage, or any other computer-readable storage medium that can be used to carry or store data.

In some embodiments, the at least one processor 32 may be composed of an integrated circuit, for example, may be composed of a single packaged integrated circuit, or may be composed of multiple integrated circuits with the same function or different functions. , including one or more central processing units (Central Processing unit, CPU), microprocessors, digital signal processors, graphics processors and combinations of various control chips. The at least one processor 32 is the control core (Control Unit) of the computer device 3, which connects various components of the entire computer device 3 through various interfaces and lines, and runs or executes programs stored in the memory 31 or module, and call the data stored in the storage 31 to execute various functions of the computer device 3 and process data, such as executing the log analysis function of the log analysis system 1 .

In some embodiments, the log parsing system 1 runs on a computer device 3 . The log parsing system 1 may include a plurality of functional modules composed of program code segments. The program codes of each program segment in the log analysis system 1 can be stored in the memory 31 of the computer device 3 and executed by at least one processor 32 to realize the log analysis function shown in FIG. 3 .

Although not shown, the computer device 3 may also include a power supply (such as a battery) for supplying power to each component. Preferably, the power supply may be logically connected to the at least one processor 32 through a power management device, thereby realizing Manage functions such as charging, discharging, and power management. The power supply may also include one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators and other arbitrary components. The computer device 3 may also include various sensors, a Bluetooth module, a Wi-Fi module, a display, etc., which will not be repeated here.

It should be understood that the embodiments are only for illustration, and are not limited by the structure in terms of the scope of the patent application.

The above-mentioned integrated units implemented in the form of software function modules can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, and includes several instructions for enabling a computer device (which may be a server, a personal computer, etc.) or a processor (processor) to execute part of the method described in each embodiment of the present application.

In a further embodiment, referring to FIG. 3 , the at least one processor 32 can execute the operating system of the computer device 3 and various installed applications (such as the log analysis system 1), program codes, etc., For example, the various modules mentioned above.

In one embodiment of the present application, the storage 31 stores one or more instructions (that is, at least one instruction), and the at least one instruction is executed by the at least one processor 32 to realize the log shown in FIG. 3 Parsing purposes.

In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical units, that is, they may be located in one place, or may also be distributed to multiple networks on the unit. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional module in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented not only in the form of hardware, but also in the form of hardware plus software function modules.

It will be apparent to those skilled in the art that the present application is not limited to the details of the exemplary embodiments described above, but that the present application can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application. Therefore, no matter from any point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the application is defined by the appended claims rather than the above description, so it is intended to All changes within the meaning and range of equivalents of the elements are embraced in this application. Any reference sign in a claim should not be construed as limiting the claim to which it relates. Furthermore, it is clear that the word "comprising" does not exclude other elements or the singular does not exclude the plural. A plurality of units or devices stated in the device claim may also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to denote names and do not imply any particular order.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application without limitation. Although the present application has been described in detail with reference to the above preferred embodiments, those of ordinary skill in the art should understand that the present application can be The technical solution shall be modified or equivalently replaced without departing from the spirit and scope of the technical solution of the present application.

1: Log parsing system 10: log access module 11: Data analysis module 12: Data processing module 13: Data output module 14:Remote control module 15:Visualization module 3: Computer device 31: Storage 32: Processor S1~S6: steps

In order to more clearly illustrate the technical solutions in the embodiments of the present application or in the prior art, the accompanying drawings that need to be used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings in the following description are only It is an embodiment of the present application, and those skilled in the art can also obtain other drawings according to the provided drawings without creative work.

FIG. 1 is a functional module diagram of a log parsing system provided by an embodiment of the present application.

FIG. 2 is a structural diagram of a computer device provided by an embodiment of the present application.

Fig. 3 is a flow chart of the log parsing method provided by the embodiment of the present application.

S1~S6: steps

Claims (10)

A log parsing system, wherein the system includes multiple modules, and the multiple modules include: Log access module, used to obtain log data from different communication protocols; The data parsing module is used to analyze the obtained log data in parallel, and obtain the log data of failed parsing and the log data of successful parsing; The data parsing module is also used to perform data structured processing on the successfully parsed log data to obtain structured data; A data processing module, configured to correct the log data that fails to be parsed and the structured data, and obtain the corrected data; The data output module is used to output the corrected data to the terminal. The log parsing system as described in claim item 1, wherein the log parsing system includes a plug-in program, and the plug-in program is used to expand the plurality of modules; The log analysis system also includes a remote control module, the remote control module is used to monitor the setting file of the log analysis system according to a predetermined time, and when an updated setting file is monitored, load The updated setting file updates and configures the multiple modules; The remote control module is also used to control the plurality of modules; The log parsing system further includes a visualization module, and the visualization module realizes configuration visualization of the log parsing system by using a web page. The log parsing system as described in claim item 2, wherein, the log access module includes a universal line reader; The remote control module controls the log access module to obtain the log data from different communication protocols based on the universal line reader, and uniformly access the log data from different communication protocols to the Describe the log parsing system. The log parsing system as described in claim 2, wherein the data parsing module includes a general line interpreter; The remote control module controls the data analysis module, analyzes the log data from different communication protocols in parallel based on the universal line interpreter, and based on the general line interpreter, analyzes the The log data that is successfully parsed is processed for the data structure. The log parsing system as described in claim 2, wherein the data processing module includes a general line interpreter; The remote control module controls the data processing module, and based on the universal line interpreter, corrects the log source data of the parsing failure and the structured data, including: deleting or replacing the parsing failure , tagging said structured data. The log parsing system according to claim 2, wherein the terminal includes a local storage area, a cloud and a console connected through a communication protocol. The log parsing system according to claim 2, wherein the remote control module includes a message receiving module, an information parser and a control library. The log parsing system according to claim 7, wherein the message receiving module is used to obtain control information for the multiple modules; The information parser is used to parse the control information to obtain a parsing result, and the parsing result includes an access control modifier; The control library is used to control the multiple modules according to the access control modifier. The log parsing system as described in claim item 2, wherein, the visualization module configures the multiple modules through the World Wide Web page, and generates the setting files of the multiple modules, so that the multiple modules The group processes the log data according to the setting file; and displays the running status of the multiple modules on the web page. A log parsing method, using the log parsing system according to any one of claim items 1 to 9, wherein the method includes: Obtain log data from different communication protocols; Analyzing the log data from different communication protocols in parallel to obtain the log data of failed parsing and the log data of successful parsing; performing data structuring processing on the successfully parsed log data to obtain structured data; Correcting the log data that fails to be parsed and the structured data to obtain the corrected data; Outputting the corrected data to a terminal; Monitor the configuration file of the log analysis system according to a predetermined time, and update and configure the log analysis system according to changes in the monitored configuration file; The configuration visualization of the above log parsing process tasks is realized by using a web page.

Images