CN110413586A - Distributed information log management method and system - Google Patents
Distributed information log management method and system Download PDFInfo
- Publication number
- CN110413586A CN110413586A CN201910717119.3A CN201910717119A CN110413586A CN 110413586 A CN110413586 A CN 110413586A CN 201910717119 A CN201910717119 A CN 201910717119A CN 110413586 A CN110413586 A CN 110413586A
- Authority
- CN
- China
- Prior art keywords
- journal file
- data
- database
- stand
- distributed information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000007726 management method Methods 0.000 title claims abstract description 36
- 238000000034 method Methods 0.000 abstract description 6
- 238000010586 diagram Methods 0.000 description 3
- 238000009434 installation Methods 0.000 description 3
- 238000012360 testing method Methods 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 230000008878 coupling Effects 0.000 description 2
- 238000010168 coupling process Methods 0.000 description 2
- 238000005859 coupling reaction Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 239000004744 fabric Substances 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
- G06F16/134—Distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a kind of distributed information log management method and system, belong to journal file management, technical problems to be solved be how distributed implementation log management and log query.Method includes the journal file of each stand-alone service system being collected based on unified requestId, and the journal file of the distribution is together in series；The journal file is stored in the database of Oriented Documents type, and building index cluster in the database, indexes the corresponding journal file of each index in cluster；It is interacted by web page with user, web page is cooperated by graphic interface and database to realize that data query and data are shown.System includes log acquisition module, storage search module and query display module.
Description
Technical field
The present invention relates to journal file management domain, specifically a kind of distributed information log management method and system.
Background technique
Check that system log is to analyze the important channel solved the problems, such as on line, usually log can be output to console by we
Or in local file, by the way that according to the log of keyword search local, this is used also in stand-alone application when checking problem
It is mode very easily and effectively, but now with the expansion of userbase, stand-alone application is difficult to that large-scale consumer is supported to visit
At this moment the business asked uses distributed framework in project development, log will record in multiple servers or file, same
A service generallys use clustered deploy(ment), and an abnormal log is all difficult to navigate to specific machine, and problem analysis may be each
Clustered machine carries out log query.It does so obviously cumbersome and time consuming.At this moment unify to carry out log management and log query
Distributed information log system seems extremely important.
How distributed implementation log management and log query, be the technical issues that need to address.
Summary of the invention
Technical assignment of the invention is against the above deficiency, a kind of distributed information log management method and system to be provided, to solve
Certainly how distributed implementation log management and the problem of log query.
In a first aspect, the present invention provides a kind of distributed information log management method, for being distributed in each stand-alone service system
Journal file in system is collected and manages, and the blog management method includes the following steps:
Collect the journal file of each stand-alone service system based on unified requestId, and by the log of the distribution
File is together in series；
The journal file is stored in the database of Oriented Documents type, and building index cluster in the database, index
The corresponding journal file of each index in cluster；
Interacted by web page with user, web page cooperated by graphic interface and database with realize data query and
Data are shown.
In the above-described embodiment, each stand-alone service system is located on its corresponding server, passes through AOP collect components
The journal file of each stand-alone service system, is uniformly stored in database for the journal file of the distribution of collection and constructs cluster rope
Draw, realize the distributed management to journal file, reduces the coupling between system；And by web interface with figure circle
Face inquiry and display data, the visual analyzing and content for realizing log content are shown.
Preferably, after collecting the journal file of each stand-alone service system based on unified requestId, to each day
Will file is filtered to obtain the journal file of specific format, and the journal file of specific format is stored in the number of Oriented Documents type
According to library.
Preferably, the corresponding data of each journal file, stores every data as a document
With database, and using JSON as the format of document sequence.
Preferably, each field in data is indexed when building indexes cluster in the database.
Second aspect, the present invention provide a kind of distributed information log management system, for being distributed in each stand-alone service system
Journal file in system is collected and manages, and the Log Administration System includes:
Log acquisition module, the log acquisition module several and with stand-alone service system correspond, each log
Acquisition module is used to collect the journal file of corresponding stand-alone service system based on unified requestId；
Search module is stored, the storage search module is Oriented Documents type database, for storing the journal file,
And for constructing index cluster, the corresponding journal file of each index in cluster is indexed；
Query display module, the display enquiry module are connect with storage search module, for passing through web page and user
Interaction, web page realize that data query and data are shown by graphic interface and the cooperation of storage search module.
Each stand-alone service system is located on its corresponding server, and log acquisition module is configured in corresponding server
On, the journal file of corresponding service system is acquired by AOP component.All journal files series connection of acquisition is database
In, while configuration index cluster in the database, the corresponding journal file of each index, user interact with web page, so as to
Journal file is inquired and checked by web page in user, wherein log file contents exhibition in a manner of pictograph
Show.
Preferably, the log acquisition module includes:
Input channel, the input channel is for acquiring corresponding stand-alone service system based on unified requestId
The journal file of system；
Filter, the filter export the log text of specific format for being filtered to the journal file of acquisition
Part；
Output channel, the output channel are used to filtered journal file being sent to storage search module.
Preferably, the corresponding data of each journal file, the storage search module for using every data as
One document, and using JSON as the format of document sequence.
Preferably, the storage search module is for each field in data to be indexed.
Of the invention a kind of distributed information log management method and system have the advantage that
1, database is uniformly stored in the journal file of distribution and constructs cluster index, realized and journal file is divided
Cloth management reduces the coupling between system；
2, it is inquired with graphical interfaces by web interface and display data, realizes the visual analyzing of log content and interior
Hold and shows.
Detailed description of the invention
It to describe the technical solutions in the embodiments of the present invention more clearly, below will be to required in being described in embodiment
The attached drawing used is briefly introduced, it should be apparent that, drawings in the following description are only some embodiments of the invention, for
For those skilled in the art, without creative efforts, it can also be obtained according to these attached drawings
His attached drawing.
The following further describes the present invention with reference to the drawings.
Attached drawing 1 is a kind of flow diagram of the distributed information log management method of embodiment 1.
Specific embodiment
The present invention will be further explained below with reference to the attached drawings and specific examples, so that those skilled in the art can be with
It more fully understands the present invention and can be practiced, but illustrated embodiment is not as a limitation of the invention, the case where not conflicting
Under, the technical characteristic in the embodiment of the present invention and embodiment can be combined with each other.
The embodiment of the present invention provides a kind of distributed information log management method and system, for solve how distributed implementation day
The technical issues of will management and log query.
Embodiment 1:
A kind of distributed information log management method of the invention, for the log text being distributed in each stand-alone service system
Part is collected and manages.
The blog management method includes the following steps:
S100, the journal file that each stand-alone service system is collected based on unified requestId, and by the distribution
Journal file is together in series；
S200, the database that the journal file is stored in Oriented Documents type, and building index cluster in the database,
Index the corresponding journal file of each index in cluster；
S300, it is interacted by web page with user, web page is cooperated by graphic interface and database to realize data
Inquiry and data are shown.
Wherein, each stand-alone service system configuration acquires the service system pair by AOP component on corresponding server
The journal file answered is filtered to obtain the journal file of specific format to each journal file, and by all specific formats
Journal file, which is together in series, to be transmitted in database.
The database is the database of Oriented Documents type, and as the storage research tool of log, each journal file is corresponding
One data uses JSON as the format of document sequence using every data as a document.Configuration index in the database
When cluster, each field of data is indexed, allows to be searched.Index cluster realizes the real-time of journal file
Distributed storage.
Based on the corresponding web page of above-mentioned database sharing, which interacts with user, as displaying query facility.
Web page is cooperated by graphic interface and database to realize that data query and data are shown, can be by structuring and non-structural
Index is added in the data of change, is capable of handling the data of magnanimity, and forms column diagram, rectilinear, scatter plot, pie chart and map accordingly
Deng.
Embodiment 2:
A kind of distributed information log management system of the invention, for the log text being distributed in each stand-alone service system
Part is collected and manages, which includes log acquisition module, storage search module and query display module.
Log acquisition module several and corresponded with stand-alone service system, each log acquisition module is configured at correspondence
Stand-alone service system server on, corresponding stand-alone service is collected based on unified requestId by AOP component
The journal file of system.
In the present embodiment, log acquisition module includes input channel, filter and output channel, input channel and efferent duct
Road is used as data flow path, and corresponding stand-alone service system is acquired based on unified requestId by AOP component
After journal file, journal file is transferred to by filter by input channel, filter is adaptation journal file and database
Certain filter formats journal file by filter, obtains the journal file of specific format to store to day
In will file；Output channel is used to filtered journal file being sent to storage search module.
Storage search module is Oriented Documents type database, for storing the journal file, and for constructing indexed set
Group indexes the corresponding journal file of each index in cluster.Specifically, each journal file is one corresponding in the database
Data are stored every data as a document, and using JSON as the format of document sequence.In creation cluster index
When, each field in data is indexed.
Display interrogation model is formed with web page, which is the specific web page based on database sharing, tool
There is graphical interfaces, is interacted by web page with user.
The web page and above-mentioned database cooperate, and support the inquiry to journal file and check, realize journal file
Visualization.
When showing journal file by the web page, column diagram, rectilinear, scatter plot, pie chart and map can be formed, with
It is intuitive to show journal file.
A kind of distributed information log management system of the invention can realize a kind of distributed information log management method.
When configuring above-mentioned Log Administration System, configuration log is installed on the corresponding server of each stand-alone service system
Acquisition module, method particularly includes:
Tool installation kit loget.tar.gz is uploaded to server and is decompressed, and entrance installation directory: cd# { dir }/
Loget creates the test profile of loget, and Edition Contains are as follows:
vim test.conf
input{
stdin{}
}
output{
stdout{
Codec=> rubydebug { }
}
}
When configuration stores search module, which is configured on a server, method particularly includes:
vim config/logsearch.yml
# configuration data path
path.data:/data/es
# configuration log path
path.logs:/data/logs/es
# configuration server address
network.host:localhost
# configuration server port
http.port:9200
Configuration executes user and catalogue:
groupadd logearch
useradd logearch-g logearch-p elasticsearch
chown-R logearch:logearch elasticsearch-2.4.0
mkdir/data/es
mkdir/data/logs/es
chown-R logearch:logearch/data/es
chown-R logearch:logearch/data/logs/es
And log acquisition module and storage search module are integrated, method particularly includes:
input{
stdin{}
}
output{
logsearch{
Hosts=> " logsearchIP:9200 "
Index=> " loget-test "
}
stdout{
Codec=> rubydebug { }
}
}
Query display module is as query display tool (kiban), when configuration querying display module, needs its installation kit
Configuration is decompressed and changes, method particularly includes:
vi config/kiban.yml
# starts port because port restricted is change default port
server.port:8601
The ip of # starting service
Server.host: " the machine ip "
The address #elasticsearch
elasticsearch.url:“http://logearchIP:9200”
Startup program:
bin/kibana
Embodiment described above is only to absolutely prove preferred embodiment that is of the invention and being lifted, protection model of the invention
It encloses without being limited thereto.Those skilled in the art's made equivalent substitute or transformation on the basis of the present invention, in the present invention
Protection scope within.Protection scope of the present invention is subject to claims.
Claims (8)
1. distributed information log management method, it is characterised in that for the journal file being distributed in each stand-alone service system into
Row is collected and management, the blog management method include the following steps:
Collect the journal file of each stand-alone service system based on unified requestId, and by the journal file of the distribution
It is together in series；
The journal file is stored in the database of Oriented Documents type, and building index cluster in the database, indexes cluster
In the corresponding journal file of each index；
It is interacted by web page with user, web page is cooperated by graphic interface and database to realize data query and data
It shows.
2. distributed information log management method according to claim 1, it is characterised in that collected based on unified requestId
After the journal file of each stand-alone service system, each journal file is filtered to obtain the journal file of specific format, it will
The journal file of specific format is stored in the database of Oriented Documents type.
3. distributed information log management method according to claim 1 or 2, it is characterised in that each journal file is corresponding
One data, every data is stored as a document and database, and using JSON as the format of document sequence.
4. distributed information log management method according to claim 3, it is characterised in that building index cluster in the database
When, each field in data is indexed.
5. distributed information log management system, it is characterised in that for the journal file being distributed in each stand-alone service system into
Row is collected and management, the Log Administration System include:
Log acquisition module, the log acquisition module several and with stand-alone service system correspond, each log collection
Module is used to collect the journal file of corresponding stand-alone service system based on unified requestId；
Search module is stored, the storage search module is that Oriented Documents type database is used in combination for storing the journal file
Cluster is indexed in building, indexes the corresponding journal file of each index in cluster；
Query display module, the display enquiry module are connect with storage search module, for being handed over by web page and user
Mutually, web page realizes that data query and data are shown by graphic interface and the cooperation of storage search module.
6. distributed information log management system according to claim 5, it is characterised in that the log acquisition module includes:
Input channel, the input channel is for acquiring corresponding stand-alone service system based on unified requestId
Journal file；
Filter, the filter export the journal file of specific format for being filtered to the journal file of acquisition；
Output channel, the output channel are used to filtered journal file being sent to storage search module.
7. distributed information log management system according to claim 5 or 6, it is characterised in that each journal file is one corresponding
Data, the storage search module are used for using every data as a document, and using JSON as the format of document sequence.
8. distributed information log management system according to claim 7, it is characterised in that the storage search module is used for will
Each field in data is indexed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910717119.3A CN110413586B (en) | 2019-08-05 | 2019-08-05 | Distributed log management method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910717119.3A CN110413586B (en) | 2019-08-05 | 2019-08-05 | Distributed log management method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110413586A true CN110413586A (en) | 2019-11-05 |
| CN110413586B CN110413586B (en) | 2023-09-22 |
Family
ID=68365741
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910717119.3A Active CN110413586B (en) | 2019-08-05 | 2019-08-05 | Distributed log management method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110413586B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111177239A (en) * | 2019-12-13 | 2020-05-19 | 航天信息股份有限公司 | Unified log processing method and system based on HDP big data cluster |
| CN111522786A (en) * | 2020-04-21 | 2020-08-11 | 中国建设银行股份有限公司 | Log processing system and method |
| CN111694793A (en) * | 2020-06-12 | 2020-09-22 | 北京金山云网络技术有限公司 | Log storage method and device and log query method and device |
| CN111858475A (en) * | 2020-07-14 | 2020-10-30 | 深圳前海移联科技有限公司 | Universal distributed log context retrieval system and method |
| CN113904913A (en) * | 2021-08-19 | 2022-01-07 | 济南浪潮数据技术有限公司 | Alarm processing method, device, equipment and storage medium based on pipeline |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106709003A (en) * | 2016-12-23 | 2017-05-24 | 长沙理工大学 | Hadoop-based mass log data processing method |
| US20180137166A1 (en) * | 2016-11-17 | 2018-05-17 | Sap Se | Database Systems Architecture Incorporating Distributed Log |
| CN109710439A (en) * | 2018-12-12 | 2019-05-03 | 百度在线网络技术（北京）有限公司 | Fault handling method and device |
-
2019
- 2019-08-05 CN CN201910717119.3A patent/CN110413586B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180137166A1 (en) * | 2016-11-17 | 2018-05-17 | Sap Se | Database Systems Architecture Incorporating Distributed Log |
| CN106709003A (en) * | 2016-12-23 | 2017-05-24 | 长沙理工大学 | Hadoop-based mass log data processing method |
| CN109710439A (en) * | 2018-12-12 | 2019-05-03 | 百度在线网络技术（北京）有限公司 | Fault handling method and device |
Non-Patent Citations (1)
| Title |
|---|
| 张振;刘俊艳;: "基于微服务架构的日志监控系统的设计与实现", 软件, no. 11 * |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111177239A (en) * | 2019-12-13 | 2020-05-19 | 航天信息股份有限公司 | Unified log processing method and system based on HDP big data cluster |
| CN111177239B (en) * | 2019-12-13 | 2023-10-10 | 航天信息股份有限公司 | Unified log processing method and system based on HDP big data cluster |
| CN111522786A (en) * | 2020-04-21 | 2020-08-11 | 中国建设银行股份有限公司 | Log processing system and method |
| CN111694793A (en) * | 2020-06-12 | 2020-09-22 | 北京金山云网络技术有限公司 | Log storage method and device and log query method and device |
| CN111858475A (en) * | 2020-07-14 | 2020-10-30 | 深圳前海移联科技有限公司 | Universal distributed log context retrieval system and method |
| CN113904913A (en) * | 2021-08-19 | 2022-01-07 | 济南浪潮数据技术有限公司 | Alarm processing method, device, equipment and storage medium based on pipeline |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110413586B (en) | 2023-09-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110413586A (en) | Distributed information log management method and system | |
| Kharlamov et al. | Semantic access to streaming and static data at Siemens | |
| CN108804513A (en) | Automatic visual analysis method for big data platform | |
| CN103218239B (en) | A kind of method supporting guiding aircraft map version incremental update version | |
| US8700671B2 (en) | System and methods for dynamic generation of point / tag configurations | |
| CN110633186A (en) | Log monitoring system for electric power metering micro-service architecture and implementation method | |
| US8676746B2 (en) | Database management system risk assessment | |
| CN110457178A (en) | A kind of full link monitoring alarm method based on log collection analysis | |
| US8812544B2 (en) | Enterprise content management federation and integration system | |
| CN102915269B (en) | Method analyzed in the general journal of a kind of B/S software system | |
| US20170060972A1 (en) | Systems and methods for processing process data | |
| CN108804497A (en) | A kind of big data analysis method based on daily record | |
| US11681707B1 (en) | Analytics query response transmission | |
| US8250113B2 (en) | Web service discovery via data abstraction model | |
| CN109766206A (en) | A kind of log collection method and system | |
| CN103917970B (en) | Keyword search of customer interest in an enterprise | |
| Serena et al. | Semantic discovery in the web of things | |
| Valdestilhas et al. | Where is my URI? | |
| US20200201867A1 (en) | Inserting annotations for application tracing | |
| CN109614553A (en) | PaaS platform for log collection | |
| US20090106294A1 (en) | Method and apparatus for employing a searchable abstraction layer over enterprise-wide searchable objects | |
| US20080222381A1 (en) | Storage optimization method | |
| US7979474B2 (en) | Search center dynamic configuration using field mappings | |
| Grotton et al. | Towards a semantic data library for the social sciences | |
| JP5887236B2 (en) | Business document processing apparatus, business document processing method, and business document processing program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20230828 Address after: No. 1036, Shandong high tech Zone wave road, Ji'nan, Shandong Applicant after: Inspur Genersoft Co.,Ltd. Address before: 250100 No. 2877 Kehang Road, Sun Village Town, Jinan High-tech District, Shandong Province Applicant before: SHANDONG INSPUR GENESOFT INFORMATION TECHNOLOGY Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |