CN110413586B - Distributed log management method and system - Google Patents
Distributed log management method and system Download PDFInfo
- Publication number
- CN110413586B CN110413586B CN201910717119.3A CN201910717119A CN110413586B CN 110413586 B CN110413586 B CN 110413586B CN 201910717119 A CN201910717119 A CN 201910717119A CN 110413586 B CN110413586 B CN 110413586B
- Authority
- CN
- China
- Prior art keywords
- log
- log files
- data
- database
- distributed
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
- G06F16/134—Distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a distributed log management method and a distributed log management system, which belong to log file management and solve the technical problem of how to realize log management and log inquiry in a distributed manner. The method comprises the steps of collecting log files of each independent service system based on the uniform requestId, and connecting the distributed log files in series; storing the log files in a document-oriented database, and constructing an index cluster in the database, wherein each index in the index cluster corresponds to one log file; the web page is interacted with a user through the web page, and the web page is matched with the database through the image interface to realize data query and data display. The system comprises a log acquisition module, a storage search module and a query display module.
Description
Technical Field
The invention relates to the field of log file management, in particular to a distributed log management method and system.
Background
Looking up system logs is an important way to analyze and solve on-line problems, normally, we will output the logs to a console or a local file, when the problems are examined, the local logs are searched according to keywords, which is a very convenient and effective way to use in stand-alone applications, but now, with the expansion of the scale of users, the stand-alone applications are difficult to support the large-scale user access service, at this time, a distributed architecture is adopted in project development, the logs can be recorded into a plurality of servers or files, the same service is usually deployed in clusters, one abnormal log is difficult to locate to a specific machine, and each cluster machine is likely to carry out log query for the analysis problems. This is obviously time consuming and laborious. At this time, a distributed log system for uniformly performing log management and log query is very important.
How to realize log management and log inquiry in a distributed manner is a technical problem to be solved.
Disclosure of Invention
The technical task of the invention is to provide a distributed log management method and a system for solving the problem of how to realize log management and log inquiry in a distributed manner.
In a first aspect, the present invention provides a distributed log management method for collecting and managing log files distributed in each independent service system, the log management method including the steps of:
collecting log files of each independent service system based on the uniform requestId, and connecting the distributed log files in series;
storing the log files in a document-oriented database, and constructing an index cluster in the database, wherein each index in the index cluster corresponds to one log file;
the web page is interacted with a user through the web page, and the web page is matched with the database through the image interface to realize data query and data display.
In the embodiment, each independent service system is located on the corresponding server, the log files of each independent service system are collected through the AOP component, the collected distributed log files are uniformly stored in the database, and the cluster index is constructed, so that the distributed management of the log files is realized, and the coupling between the systems is reduced; and the data is queried and displayed through a web interface by a graphical interface, so that the visual analysis and content display of the log content are realized.
Preferably, after collecting the log files of each independent service system based on the uniform requestId, filtering each log file to obtain a log file with a specific format, and storing the log file with the specific format in a document-oriented database.
Preferably, each log file corresponds to one piece of data, each piece of data is stored as a document and is stored in a database, and JSON is used as a format for serializing the documents.
Preferably, each field in the data is indexed when an index cluster is built in the database.
In a second aspect, the present invention provides a distributed log management system for collecting and managing log files distributed in respective independent service systems, the log management system comprising:
the log acquisition modules are in one-to-one correspondence with the independent service systems, and each log acquisition module is used for collecting log files of the corresponding independent service system based on the uniform requestId;
the storage search module is a document-oriented database and is used for storing the log files and constructing an index cluster, and each index in the index cluster corresponds to one log file;
and the query display module is connected with the storage search module and is used for interacting with a user through a web page, and the web page is matched with the storage search module through an image interface so as to realize data query and data display.
Each independent service system is positioned on a corresponding server, and the log acquisition module is configured on the corresponding server and acquires log files of the corresponding service system through the AOP component. All the collected log files are connected in series in the database, meanwhile, an index cluster is configured in the database, each index corresponds to one log file, and a user interacts with the web page so as to conveniently inquire and view the log files through the web page, wherein the content of the log files is displayed in a graphic and text mode.
Preferably, the log collection module includes:
the input pipeline is used for collecting log files of the independent service systems corresponding to the unified requestId;
the filter is used for filtering the collected log files and outputting log files in a specific format;
and the output pipeline is used for transmitting the filtered log files to the storage search module.
Preferably, each log file corresponds to one piece of data, and the storage search module is used for taking each piece of data as one document and taking JSON as a document serialization format.
Preferably, the storage search module is configured to index each field in the data.
The distributed log management method and system of the invention have the following advantages:
1. the distributed log files are uniformly stored in a database and cluster indexes are constructed, so that the distributed management of the log files is realized, and the coupling between systems is reduced;
2. and inquiring and displaying data through a web interface by using a graphical interface, and realizing visual analysis and content display of log content.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the description of the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
The invention is further described below with reference to the accompanying drawings.
Fig. 1 is a flowchart of a distributed log management method according to embodiment 1.
Detailed Description
The invention will be further described with reference to the accompanying drawings and specific examples, so that those skilled in the art can better understand the invention and implement it, but the examples are not meant to limit the invention, and the technical features of the embodiments of the invention and the examples can be combined with each other without conflict.
The embodiment of the invention provides a distributed log management method and a distributed log management system, which are used for solving the technical problem of how to realize log management and log inquiry in a distributed manner.
Example 1:
the invention relates to a distributed log management method which is used for collecting and managing log files distributed in each independent service system.
The log management method comprises the following steps:
s100, collecting log files of each independent service system based on a unified requestId, and connecting the distributed log files in series;
s200, storing the log files in a document-oriented database, and constructing an index cluster in the database, wherein each index in the index cluster corresponds to one log file;
s300, interacting with a user through a web page, and matching the web page with a database through an image interface to realize data query and data display.
Each independent service system is configured on a corresponding server, the corresponding log files of the service system are collected through an AOP component, each log file is filtered to obtain log files with specific formats, and all log files with specific formats are connected in series and transmitted to a database.
The database is a document-oriented database, is used as a storage search tool of logs, corresponds to one piece of data, takes each piece of data as a document, and uses JSON as a document serialization format. When an index cluster is configured in a database, each field of data is indexed so that it can be searched. The index cluster realizes the real-time distributed storage of the log files.
And constructing a corresponding web page based on the database, wherein the web page interacts with a user and is used as a presentation query tool. The web page is matched with the database through the image interface to realize data query and data display, structured and unstructured data can be added into an index, massive data can be processed, and a bar graph, a linear graph, a scatter graph, a pie graph, a map and the like are formed according to the data.
Example 2:
the invention relates to a distributed log management system which is used for collecting and managing log files distributed in each independent service system.
The system comprises a plurality of log acquisition modules and independent service systems, wherein each log acquisition module is configured on a server of the corresponding independent service system, and collects log files of the corresponding independent service system based on the uniform requestId through an AOP component.
In this embodiment, the log collection module includes an input pipeline, a filter and an output pipeline, where the input pipeline and the output pipeline are both used as data flow channels, after the AOP component collects log files of the corresponding independent service system based on the uniform requestId, the log files are transmitted to the filter through the input pipeline, the filter is a specific filter adapted to the log files and the database, and format conversion is performed on the log files through the filter, so as to obtain log files in a specific format for storage in the log files; the output pipeline is used for transmitting the filtered log files to the storage search module.
The storage search module is a document-oriented database and is used for storing the log files and constructing an index cluster, and each index in the index cluster corresponds to one log file. Specifically, in the database, each log file corresponds to one piece of data, each piece of data is stored as a document, and JSON is used as a document serialization format. When creating a cluster index, each field in the data is indexed.
The display query model is formed with web pages, which are specific web pages built based on a database, with graphical interfaces through which users interact.
The web page is matched with the database to support the inquiry and the viewing of the log file, so that the visualization of the log file is realized.
When the log file is displayed through the web page, a column diagram, a straight line diagram, a scatter diagram, a pie chart and a map can be formed so as to intuitively display the log file.
The distributed log management system can realize a distributed log management method.
When the log management system is configured, a configuration log acquisition module is installed on a server corresponding to each independent service system, and the specific method comprises the following steps:
uploading a tool installation package log.tar.gz to a server, decompressing and entering an installation catalog: cd# { dir }/log, creating a test configuration file of the log, and editing the contents as follows:
vim test.conf
input{
stdin{}
}
output{
stdout{
codec＝>rubydebug{}
}
}
when a storage search module is configured, the storage search module is configured on a server, and the specific method comprises the following steps:
vim config/logsearch.yml
configuration data path
path.data:/data/es
Configuration log path
path.logs:/data/logs/es
# configuration server address
network.host:localhost
# configuration server port
http.port:9200
Configuration execution users and directories:
groupadd logearch
useradd logearch-g logearch-p elasticsearch
chown-R logearch:logearch elasticsearch-2.4.0
mkdir/data/es
mkdir/data/logs/es
chown-R logearch:logearch/data/es
chown-R logearch:logearch/data/logs/es
the method comprises the following steps of:
input{
stdin{}
}
output{
logsearch{
hosts＝>"logsearchIP:9200"
index＝>"loget-test"
}
stdout{
codec＝>rubydebug{}
}
}
the query display module is used as a query display tool (kiman), and when the query display module is configured, the installation package of the query display module needs to be decompressed and the configuration is changed, and the concrete method is as follows:
vi config/kiban.yml
# Start Port changes default Port because Port is restricted
server.port:8601
Ip for # initiation service
server.host: "native ip"
# elastic search address
elasticsearch.url:“http://logearchIP:9200”
The start-up procedure:
bin/kibana
the above-described embodiments are merely preferred embodiments for fully explaining the present invention, and the scope of the present invention is not limited thereto. Equivalent substitutions and modifications will occur to those skilled in the art based on the present invention, and are intended to be within the scope of the present invention. The protection scope of the invention is subject to the claims.
Claims (2)
1. The distributed log management method is characterized by being used for collecting and managing log files distributed in each independent service system, and comprises the following steps:
collecting log files of each independent service system based on the uniform requestId, filtering each log file to obtain log files with specific formats, storing the log files with the specific formats in a document-oriented database, and transmitting all the log files with the specific formats to the database in series;
storing the log files in a document-oriented database, wherein each log file corresponds to one piece of data, storing each piece of data as a document and the database, taking JSON as a document serialization format, constructing an index cluster in the database, indexing each field in the data, and each index in the index cluster corresponds to one log file;
the web page is interacted with a user through the web page, and the web page is matched with the database through the image interface to realize data query and data display.
2. A distributed log management system for collecting and managing log files distributed among individual service systems, the log management system comprising:
the log acquisition modules are in one-to-one correspondence with the independent service systems, and each log acquisition module is used for collecting log files of the corresponding independent service system based on the uniform requestId;
the storage search module is a document-oriented database and is used for storing the log files and constructing an index cluster, and each index in the index cluster corresponds to one log file;
the inquiry display module is connected with the storage search module and is used for interacting with a user through a web page, and the web page is matched with the storage search module through an image interface so as to realize data inquiry and data display;
the log acquisition module comprises:
the input pipeline is used for collecting log files of the independent service systems corresponding to the unified requestId;
the filter is used for filtering the collected log files and outputting log files in a specific format;
the output pipeline is used for transmitting the filtered log files to the storage search module;
each log file corresponds to one piece of data, and the storage search module is used for taking each piece of data as a document and taking JSON as a document serialization format;
the storage search module is used for indexing each field in the data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910717119.3A CN110413586B (en) | 2019-08-05 | 2019-08-05 | Distributed log management method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910717119.3A CN110413586B (en) | 2019-08-05 | 2019-08-05 | Distributed log management method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110413586A CN110413586A (en) | 2019-11-05 |
| CN110413586B true CN110413586B (en) | 2023-09-22 |
Family
ID=68365741
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910717119.3A Active CN110413586B (en) | 2019-08-05 | 2019-08-05 | Distributed log management method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110413586B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111177239B (en) * | 2019-12-13 | 2023-10-10 | 航天信息股份有限公司 | Unified log processing method and system based on HDP big data cluster |
| CN111522786A (en) * | 2020-04-21 | 2020-08-11 | 中国建设银行股份有限公司 | Log processing system and method |
| CN111694793A (en) * | 2020-06-12 | 2020-09-22 | 北京金山云网络技术有限公司 | Log storage method and device and log query method and device |
| CN111858475A (en) * | 2020-07-14 | 2020-10-30 | 深圳前海移联科技有限公司 | Universal distributed log context retrieval system and method |
| CN113904913B (en) * | 2021-08-19 | 2024-10-18 | 济南浪潮数据技术有限公司 | Method, device, equipment and storage medium for alarm processing based on pipeline |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106709003A (en) * | 2016-12-23 | 2017-05-24 | 长沙理工大学 | Hadoop-based mass log data processing method |
| CN109710439A (en) * | 2018-12-12 | 2019-05-03 | 百度在线网络技术（北京）有限公司 | Fault handling method and device |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10474668B2 (en) * | 2016-11-17 | 2019-11-12 | Sap Se | Database systems architecture incorporating distributed log |
-
2019
- 2019-08-05 CN CN201910717119.3A patent/CN110413586B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106709003A (en) * | 2016-12-23 | 2017-05-24 | 长沙理工大学 | Hadoop-based mass log data processing method |
| CN109710439A (en) * | 2018-12-12 | 2019-05-03 | 百度在线网络技术（北京）有限公司 | Fault handling method and device |
Non-Patent Citations (1)
| Title |
|---|
| 基于微服务架构的日志监控系统的设计与实现;张振;刘俊艳;;软件(11);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110413586A (en) | 2019-11-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110413586B (en) | Distributed log management method and system | |
| Paris et al. | NilmDB: The non-intrusive load monitor database | |
| US8700671B2 (en) | System and methods for dynamic generation of point / tag configurations | |
| CN108804513A (en) | Automatic visual analysis method for big data platform | |
| CN110457178A (en) | A kind of full link monitoring alarm method based on log collection analysis | |
| CN110659790A (en) | Experiment management and data analysis system | |
| EP2037379A1 (en) | System and method for performing a file system operation on a specified storage tier | |
| US20070192473A1 (en) | Performance information monitoring system, method and program | |
| JP2005062941A (en) | Method for analyzing performance information | |
| CN112100138A (en) | Log query method and device, storage medium and electronic equipment | |
| CN115269515B (en) | Processing method for searching specified target document data | |
| CN111124839A (en) | Distributed log data monitoring method and device | |
| US20080222381A1 (en) | Storage optimization method | |
| CN202535378U (en) | Test data processing apparatus | |
| CN112486985A (en) | Boiler data query method, device, equipment and storage medium | |
| CN117093555A (en) | Method, device, equipment and readable storage medium for acquiring equipment state information | |
| CN116185677A (en) | Automatic fault positioning method, system and medium | |
| CN111324688A (en) | Semi-structured data and unstructured data acquisition system based on events | |
| CN116708505A (en) | Intelligent mine data acquisition method and equipment | |
| CN115617623A (en) | Performance index data visualization method and device, electronic equipment and storage medium | |
| CN109684158B (en) | State monitoring method, device, equipment and storage medium of distributed coordination system | |
| CN113014661A (en) | Log acquisition and analysis method for distributed system | |
| JP2018147350A (en) | Apparatus for analyzing actual use of information processing system, and method for analyzing actual use | |
| CN113360472A (en) | Multidimensional data analysis method and system based on Kylin OLAP | |
| Podhorszki et al. | Presentation and analysis of grid performance data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20230828 Address after: No. 1036, Shandong high tech Zone wave road, Ji'nan, Shandong Applicant after: Inspur Genersoft Co.,Ltd. Address before: 250100 No. 2877 Kehang Road, Sun Village Town, Jinan High-tech District, Shandong Province Applicant before: SHANDONG INSPUR GENESOFT INFORMATION TECHNOLOGY Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |