WO2012155455A1

WO2012155455A1 - Log analysis method and system based on web platform

Info

Publication number: WO2012155455A1
Application number: PCT/CN2011/081062
Authority: WO
Inventors: 张立
Original assignee: 中兴通讯股份有限公司
Priority date: 2011-05-13
Filing date: 2011-10-20
Publication date: 2012-11-22
Also published as: CN102780726B; CN102780726A

Abstract

A log analysis method based on WEB platform is provided by the invention. The method comprises: a log collection module periodically collects logs created from each device on the network, the log collection module sends the collected logs to a log server, and the log server performs index and statistic classification to obtain the index data and the statistic classification data. Also, a log analysis system based on WEB platform is provided by the invention. By providing an integrated log convergence process platform for collection, search, browse, analysis, backup, and download log, the invention solves the problem in the prior art that the logs of each device only are simply collected, there is not a integrated scheme for collection, search, browse, analysis, backup, and download log.

Description

Log analysis method and system based on WEB platform

The invention relates to log management, in particular to a log collection, analysis and search method and system based on a network (WEB) platform. Background technique

Currently, the service logs, operating system logs, and database logs of each device on the network are backed up separately. When viewing logs, you need to connect to each device to obtain logs, and then manually analyze the log contents. This type of processing seriously affects the maintenance staff's work efficiency and the rapid location and elimination of faults. Moreover, in this mode of work, it is difficult to perform statistical statistics and auditing on a large amount of log information.

In the existing environment, there are some tools for simple collection of logs of each device in the network, but there is no complete solution for collecting, searching, browsing, analyzing, backing up, and downloading logs. Summary of the invention

The object of the present invention is to provide a log analysis method and system based on the WEB platform, and solve the following technical problems: 1. The maintenance personnel are not efficient, and the fault is difficult to locate and eliminate quickly; 2. Statistics on a large amount of log information It is more difficult to audit; third, it is simply collecting the logs of each device, and there is no complete solution for collecting, searching, browsing, analyzing, backing up, and downloading logs.

According to an aspect of the present invention, a log analysis method based on a WEB platform is provided, and the method includes the following steps:

The log collection module periodically collects logs generated by each device on the network, and collects the collected logs. The log is uploaded to the log server;

The log server indexes and statistically classifies the logs to obtain index data and statistical classification data.

Preferably, after the log server obtains the index data and the statistical classification data, the method further includes: searching, by the log server, the log that matches the search condition from the index data according to a search condition input by the user.

Preferably, the log server performs indexing and statistical categorization of the log to obtain index data and statistical categorization data, and further includes: the log server compressing and backing the log.

The step log collection module periodically collects logs generated by each device on the network, including:

The log server establishes a connection with each device on the network through the network;

The log collection module installed on the device periodically sends a list of log files generated by the device to the log server.

The log server selects a log name to be collected from the log file list and sends the log name to the device;

The log collection module collects logs according to the selected log name.

The uploading the collected log to the log server includes:

The log collection module uploads the collected log to the log server by using FTP or SYSLOG.

The log server saves the collected logs.

The log server indexes and statistically classifies the log, and obtains index data and statistical classification data, including: the log server according to the file name, time, level, error code, and log content Indexing the collected logs to obtain index data; the log server respectively according to four classification criteria: file name, time, level, and error code The collected logs are classified and counted to obtain classified statistical data.

According to another aspect of the present invention, a log analysis system based on a WEB platform is provided, the system comprising:

The log collection module is configured to periodically collect logs generated by each device on the network, and upload the collected logs to the log server.

The log analysis module is configured to index and statistically classify the logs uploaded to the log server, and obtain index data and statistical classification data.

Preferably, the system further includes: a log search module, configured to search for a log that matches the search condition from the index data obtained by the log analysis module; and a log backup download module configured to compress the log uploaded to the log server Backup.

Preferably, the log collection module periodically uploads the collected logs to the log server by means of FTP or SYSLOG.

Preferably, the log analysis module is configured to index the logs uploaded to the log server according to five analysis domains: file name, time, level, error code, and log content, to obtain index data;

The log analysis module is configured to perform classification and statistics on the logs uploaded to the log server according to four classification criteria: file name, time, level, and error code, to obtain classified statistical data.

Compared with the prior art, the beneficial effects of the present invention are as follows: 1. The maintenance staff's work efficiency is improved, so that the fault can be quickly located and eliminated; 2. The large amount of log information can be counted and audited; Provides a simple collection log, while providing a complete solution for searching, browsing, analyzing, backing up, and downloading. DRAWINGS

1 is a flowchart of a log analysis method based on a WEB platform according to an embodiment of the present invention; FIG. 2 is a structural structure of a log analysis system based on a WEB platform according to an embodiment of the present invention; Schematic. detailed description

The preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings.

FIG. 1 is a flowchart of a log analysis method based on a WEB platform according to an embodiment of the present invention. As shown in FIG. 1, the method includes the following steps:

Step 101, device configuration.

The device configuration is completed by the user through the WEB management platform of the log system. The device information that is entered includes the description information of the device, the parameter information that needs to be collected, and so on. Users can view all device information in a list on the management platform.

The device information is as shown in Table 1 below:

Table 1

Step 102, log collection.

There are two ways to collect logs: File Transfer Protocol (FTP) and System Log (SYSLOG, System Log). These two methods can be switched through system configuration.

The process of collecting the FTP mode is as follows: The log server communicates with the agent that is installed on the device, that is, the log collection module, and obtains a list of log files on the device through the agent. Then, the log server filters the log file to be collected, and sends the file name to the agent. The agent calls the put command of the FTP client in the operating system to upload the log file to the log server. The log server saves the collected log files by folder, and a device generates a folder. The FTP collection mode initiates the collection task by the log server. Therefore, you need to avoid the repeated collection of logs. The log server identifies the collection status of the log file by the log file name and the last modification time of the log file. After the collection is complete, the log server saves the information into the database. This information is used to determine whether the log file has been collected before the next acquisition. If the modification time of the log file on the device is the same as the last modification time of the log file in the database, the configuration is not collected. Otherwise, the collection status information of the log file is saved and saved in the database. The log file name in the log directory is deleted. Corresponding files, and save the newly collected log files in the corresponding folder of the device in the log directory.

In the SYSLOG mode, the device actively reports the log content. After the log server is running, the SYSLOG service is automatically opened. Each device actively uploads logs to the log server through the SYSLOG client. The SYSLOG collection is saved in the same way as FTP. However, since each device only uploads specific log contents, the file name is generated by the log server. Therefore, there is no log duplication.

Step 103, log analysis.

There are two tasks in log analysis: one is to index the specific log content to facilitate quick search of the log content, and the other is to classify the log content according to certain classification criteria, such as log level, log time and other classification criteria. .

Log indexing and searching is based on the open source Lucene search tool library, which uses Lucene's built-in standard parser to analyze, index, and search log content.

The Zhizhi Index contains a total of five analysis domains, as shown in Table 2 below:

Name Type Description

File name save the name of the log file

Time to save the print time of the specific log Level save log level

Error code Save the error code of the log, you can customize the content of the log analysis, save the log content, need to analyze and save, you can search according to the keywords in the log

Table 2

After the logs are categorized according to certain classifications, the statistics are saved into the database. From the WEB management platform of the log system, users can see information such as icons of statistical information, such as histograms and pie charts.

Step 104, log search.

The user can search the logs according to various conditions through the log management system of the WEB management platform. The search conditions are displayed on the page in the form of a form, and the user selects different combinations of conditions to search according to the specific situation. The log server retrieves from the index data based on the retrieval conditions entered by the user. The log search results are presented to the user on the page in a list. Since the amount of logs is very large, showing all the data is meaningless in most cases, so the log system is paged in reverse order of time and does not exceed the maximum value of the integer (INT) type. The search criteria that the user can select are as shown in Table 3 below:

Name Description

Log file search in which log files need to be performed

Start time Search in the log printed after the start time End time Search in the log printed before the end time Error code Search according to the specific error code of the log

Log level Search based on the specific level of the log

The keyword searches the log content for content that matches the keyword. The keyword is analyzed by the analyzer and searched according to the pre-match plus 80% similarity fuzzy matching. table 3

The user can browse the context information of the log by clicking the specific log entry in the search list. The number of browsing rows can be configured through the system. The clicked log entry will be highlighted in the log context.

Step 105, log backup.

The system uses scheduled tasks to back up according to the configured backup cycle. Backups are classified by device, and the compression method is zip. After the backup, the user can browse all the backup files through the WEB management platform of the log system, and can download, delete, and browse the basic information of the backup log files.

2 is a schematic structural diagram of a log analysis system based on a WEB platform according to an embodiment of the present invention. As shown in FIG. 2, the system includes a log collection module 1, a log analysis module 2, a log search module 3, and a log backup download module. The storage module 5, the device management module 6, the system configuration module 7, the user management module 8, the web management platform 9, the interface module 10, the device 11, and the log server 12. The log collection module 1 is configured to collect logs of each device; the log analysis module 1 is configured to analyze the collected logs; the log search module 3 is configured to search logs according to the retrieval conditions; and the log backup download module 4 is configured to download logs and back up; The storage module 5 is configured to store system configuration information, the collection information and the log; the device management module 6 is configured to configure and manage device information; the system configuration module 7 is configured to configure system information; and the user management module 8 is configured to manage the use of the log system. The WEB management platform 9 is used for the user to operate the log system; the interface module 10 is used for the log server 12 to communicate with the log collection module 1 on the device 11; the device 11 is used to generate logs, and the device 11 can be multiple; Server 12 is used to store logs and run various software programs.

When the system works, first deploy the WEB application. This system deploys WEB applications based on TOMCAT application server. The log analysis module 2, the log search module 3, the log backup download module 4, the storage module 5, the device management module 6, the system configuration module 7, the user management module 8, the web management platform 9, and the interface module 10 are stored and run on the log server 12. on. Log collection module 1 It is put on and operated on the device 11, and the device 11 can be multiple.

The user configures the system through the system configuration module 7. The system deploys a WEB application based on the TOMCAT application server. After logging in to the WEB management platform 9 and entering the system configuration page, the system configuration module 7 configures the system, such as setting the log storage directory, the log system collection interval, and the log collection mode. Users can also use the system default configuration.

The user completes device information entry through the device management module 6. The user logs in to the WEB management platform 9, and enters the device management page, enters the device information and saves it. The specific device information includes device name, IP address, user name, operating system, and log directory. The device name is used to indicate the name of the device that needs to be managed uniformly. The IP address is the IP address of the device 11 in the network. When the log server 12 collects the logs on the device 11, it needs to know the specific IP address of the device 11 to connect. The username is the username of the login device operating system. The operating system is the operating system type of the device 11, such as a Windows system or a Linux system. The log directory is a directory in which the logs to be collected are stored on the device 11, and the directory can be multiple. After the user has configured the device information, all the device information can be viewed in a list on the WEB management platform 9.

The user collects logs on the device periodically through the log collection module.

There are two ways to collect logs: FTP and SYSLOG. These two methods can be switched through system configuration.

The process of collecting the FTP mode is as follows: The interface module 10 periodically communicates with the log collection module 2, which is installed on the device 11, by the device IP, and obtains a log file list on the device 11 through the agent. The interface module 10 then filters the log file to be collected, and sends the file name to the agent of the device 11, and the agent invokes the operating system command to actively upload the log file to the log server 12 through the PUT mode of the FTP. The interface module 10 divides the collected log files into files in the log storage directory set by the system. The folder is saved, and a device generates a folder. The FTP collection mode is initiated by the log server 12, so it is necessary to avoid repeated collection of logs. The interface module 10 identifies the collection state of the log file by the log file name and the last modification time of the log file. After the acquisition is complete, save the information into the database. Before this next acquisition, use this information to determine whether the log file has been collected. If the modification time of the log file on the device 11 is the same as the last modification time of the log file in the database, the configuration is not collected. Otherwise, the collection status information of the log file is saved and saved in the database. The file corresponding to the file name, and save the newly collected log file in the folder corresponding to the device in the log save directory.

The process of the SYSLOG mode is as follows: After the log system is running, if the collection mode set by the system configuration module 7 is SYSLOG mode, the log server 12 automatically opens the SYSLOG service, and the device 11 actively uploads logs to the log server 12 through the SYSLOG client. 10 Save the log in the folder corresponding to the device in the log save directory. The SYSLOG collection is saved in the same way as FTP. Since the device only uploads specific log content, the file name is generated by the log server 12 and stored in the database. Therefore, there is no problem of repeated log collection.

The user periodically analyzes the log through the log analysis module 2. The log analysis module 2 reads the log content collected by the log collection module 2, performs statistical classification and indexing on the log content of the fixed format to provide a quick search. The above work is performed by the system timer at a certain period.

The log analysis module 1 mainly performs two tasks: one is to index the specific log content to facilitate quick search of the log content, and the other is to classify the log content according to certain classification criteria.

The log index is developed based on the open source Lucene search tool library, and the log content is analyzed and indexed using Lucene's built-in standard analyzer. The log index contains a total of five analysis domains, namely file name, time, level, error code, and log content. Where the classes of the first four analysis domains Types are all "save", and the type of log content analysis field is "Analysis, Save". The Lucene search tool library generates a log index based on the type of analysis domain and analysis domain.

Log classification statistics are information statistics of log data according to a certain classification. For example, according to the classification criteria such as the log level or the log time, the log data is statistically collected, and the obtained log classification statistical information is saved into the database.

After the log analysis module 2 performs the log index and log classification statistics for one cycle, the user can view the statistical analysis report of the log through the WEB management platform 9. The user enters the analysis statistics page of the WEB management platform 9 and inputs the analysis and statistics conditions. The WEB management platform 9 calls the log analysis module 2, and the log analysis module 2 reads the previously saved classification statistical information or index information from the storage module 5, and uses the data. The form is displayed in the form of a chart, such as a histogram or a pie chart.

The user searches the log through the log search module 3 based on the search criteria. The user can perform a quick search of the log according to various search conditions through the WEB management platform 9. The search conditions are displayed on the page in the form of a form, and the user selects different combinations of conditions to search according to the specific situation. The log search module 3 retrieves a log matching the search condition from the index data based on the search condition input by the user. The log search results are presented to the user on the page in a list. Since the volume of the log is very large, displaying all the data is meaningless in most cases, so the log search module 3 displays the pages in reverse order of time, and the maximum does not exceed the maximum value of the INT type. The user can browse the context information of the log by clicking the specific log entry in the search list. The number of browsing rows can be configured through the system configuration module. The clicked log entry will be highlighted in the log context.

The search criteria that the user can select are shown in the following table: log file, start time, end time, error code, log level, keyword. The log file indicates which log files need to be searched; the start time is searched in the log printed after the start time; the end time is searched in the log printed before the end time; the error code is based on the log specific The error code is searched; the log level is searched according to the specific level of the log; the keyword searches for the content matching the keyword in the log content, and the keyword is analyzed by the analyzer and fuzzy matching is matched according to the previous match plus 80% similarity Ways to search.

Users can also view the error log quick solution through log search module 3. The user logs in to the search page of the WEB management platform 9 and enters the search criteria and submits the search log. If the search log is an error log, you can also click to view the error log quick solution to monitor and troubleshoot the device 11 in the network.

The user downloads the module through the log backup 4 to back up and manage the backup file. The log backup download module 4 sorts and compresses the log files in the log save directory according to the configured backup cycle. The compression mode is zip. After the backup, the user can browse all the backup files through the WEB management platform 9, and can download, delete, and browse the basic information of the backup log files.

In summary, the present invention solves the following problems by providing a complete log centralized processing platform for log collection, search, browsing, analysis, backup, and download: The maintenance log personnel are not efficient, and faults are difficult to locate quickly. And troubleshooting; it is difficult to collect and audit a large amount of log information; it is simply collecting the logs of each device, and there is no complete solution for collecting, searching, browsing, analyzing, backing up, and downloading logs.

Although the invention has been described in detail above, the invention is not limited thereto, and various modifications may be made by those skilled in the art in accordance with the principles of the invention. Therefore, modifications made in accordance with the principles of the present invention should be construed as falling within the scope of the present invention.

Claims

Claim

A log analysis method based on a WEB platform, wherein the method comprises the following steps:

The log collection module periodically collects logs generated by each device on the network, and uploads the collected logs to the log server.

2. The method according to claim 1, wherein, after the log server obtains the index data and the statistical classification data, the method further includes:

The log server searches for the log that matches the search condition from the index data according to a search condition input by the user.

3. The method according to claim 1, wherein the log server indexes and statistically classifies the log, and obtains index data and statistical classification data, and further includes:

The log server performs a compressed backup of the log.

4. The method according to claim 1, wherein the log collection module periodically collects logs generated by each device on the network, including:

The log collection module installed on the device periodically sends a log file list generated by the device to the log server.

The log collection module collects logs according to the selected log name.

The method of claim 1, wherein the uploading the collected log to the log server comprises: the log collecting module collecting the collected log by using a file transfer protocol FTP or a system log SYSLOG Uploaded to the log server; the log service The server saves the collected logs.

The method according to claim 1, wherein the log server indexes and statistically classifies the log, and obtains index data and statistical classification data, including:

The log server indexes the collected logs according to the file name, time, level, error code, and log content, and obtains index data.

The log server separately classifies and collects the collected logs according to four classification criteria: file name, time, level, and error code, and obtains classified statistical data.

A log analysis system based on the WEB platform, wherein the system includes: a log collection module, configured to periodically collect logs generated by each device on the network, and upload the collected logs to a log server;

8. The system according to claim 7, wherein the system further comprises:

The search module is configured to search for a log matching the search condition from the index data obtained by the log analysis module;

The log backup download module is configured to perform a compressed backup on the log uploaded to the log server.

9. The system according to claim 7, wherein the log collection module periodically uploads the collected log to the log server by means of FTP or SYSLOG.

10. The system according to claim 7, wherein the log analysis module is configured to index the logs collected by the upload to the log server according to five analysis domains: file name, time, level, error code, and log content. , get index data;

The log analysis module is configured to perform classification and statistics on the logs collected by the uploading to the log server according to four classification criteria: file name, time, level, and error code, to obtain classified statistical data.