CN105260452A - System and method for collecting, searching, and analyzing offline log - Google Patents
System and method for collecting, searching, and analyzing offline log Download PDFInfo
- Publication number
- CN105260452A CN105260452A CN201510657081.7A CN201510657081A CN105260452A CN 105260452 A CN105260452 A CN 105260452A CN 201510657081 A CN201510657081 A CN 201510657081A CN 105260452 A CN105260452 A CN 105260452A
- Authority
- CN
- China
- Prior art keywords
- information
- daily record
- log
- file
- changed
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Quality & Reliability (AREA)
- Computing Systems (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a system and method for collecting, searching, and analyzing an offline log. A push apparatus acquires a log on a node server, and pushes the log to a central server in real time. A monitoring apparatus monitors a general log list on the central server, and writes a path and category label information of a changed log file into a message queue. A relocating apparatus reads information which comprises information of the changed log file from the message queue, and relocates contents of the changed log file in a file corresponding to the changed log file in classification. A log collection and dividing apparatus divides a read file, and sends information after the division to a log storing and searching apparatus. The log storing and searching apparatus stores received information and performs classification indexing on received information. A data presentation apparatus presents found data. According to the system and method for collecting, searching, and analyzing an offline log provided by the invention, real-time collection, analysis and data presentation of offline logs of different regions are realized, and a problem processing time is saved.
Description
Technical field
The present invention relates to field of computer technology, particularly a kind of offline logs collects inquiry and analysis system and method.
Background technology
Along with the development of computer technology, large data age arrives, and the daily record that program, system etc. produce is more and more important, in a pile journal file, more needs to carry out centralized management and statistical study.Traditional log collection and analytical approach inquire specified file according to different journal file names, then carry out Collection and analysis to file.But, in time needing the number of machines of checking very many, how fast finding, analyze daily record, become more and more important.Meanwhile, in the face of when corporate business to be very wide, daily record is distributed on the server of different geographical, and due to the privacy of business, server only has the extranet access of minority loom energy, and at this moment, how to carry out confluence analysis to these daily records is a great problem equally.
Summary of the invention
For above-mentioned technical matters, the invention provides a kind of save labour, cost low, can quick positioning question place, the offline logs of simultaneously greatly having saved the issue handling time collects inquiry and analysis system and method.
For solving the problems of the technologies described above, the technical solution used in the present invention is: provide a kind of offline logs to collect inquiry and analysis system, comprise pusher, supervising device, replay device, log collection cutter sweep, daily record memory scan device and data display device;
Described pusher for obtaining the daily record on node server, and with the form real time propelling movement of class discrimination on central server;
Described supervising device is for monitoring the daily record composite catalog on central server, and the journal file in described daily record composite catalog is when changing, the path of the journal file changed and class label information are used in the message queue of storing data information as an information write;
Described replay device is used for from described message queue, read the information comprising the log-file information changed, and the content of the journal file changed is refitted in a unique file corresponding with the journal file classification changed;
Described log collection cutter sweep is used for reading the file after resetting from described replay device, carries out cutting, and the information after cutting is sent to described daily record memory scan device with uniform data structure stores the file read;
Described daily record memory scan device is used for storing the information after the format received, and carries out category index;
The api that described data display device calls described daily record memory scan device retrieves, and the data retrieved is shown.
The present invention is owing to adopting above technical scheme, its technique effect reached is: the present invention well achieves real-time collecting to a large amount of offline logs in zones of different, carries out unified log analysis and data exhibiting, effective by daily record centralized management, problem investigation and positioning problems are provided and operates easily, greatly save the time of issue handling, no longer need to log in corresponding server to check daily record and problem analysis, saved labour and production cost to a great extent.
More preferably, in technique scheme, described pusher is multiple, is arranged on respectively on corresponding node server.
The beneficial effect of above-mentioned further scheme is adopted to be: multiple pusher is arranged on corresponding node server, achieve the real time propelling movement of monitoring to numerous node server and log information, do not need to re-recognize for the server of login correspondence carry out inquiry log information, saved labour.
More preferably, in technique scheme, described pusher carries out slicing treatment to the daily record on node server as required, and up-to-date section is sent to central server.
Adopt the beneficial effect of above-mentioned further scheme to be: to carry out slicing treatment to daily record, improve the speed that log information pushes to a certain extent, improve the efficiency of problem investigation.
More preferably, in technique scheme, data structure is json structure after reunification.
Adopt the beneficial effect of above-mentioned further scheme to be: to adopt unified json structure to carry out the write of data, make the data that write more regular, and also more convenient when information reads, also more targeted.
More preferably, in technique scheme, the data message got is shown with the form of Web by described data display device.
Adopt the beneficial effect of above-mentioned further scheme to be: the data message got is carried out Web displaying, facilitate the analysis and solve of technician to the problem types of the position (server name or IP) of the node server of the problem of generation and generation.
More preferably, in technique scheme, described supervising device is arranged on central server.
The beneficial effect of above-mentioned further scheme is adopted to be: supervising device is arranged on central server, facilitates monitoring node server being sent to information on central server.
More preferably, in technique scheme, when described log collection cutter sweep carries out data processing, that employing is logstash or fluentd.
Adopt the beneficial effect of above-mentioned further scheme to be: logstash or fluentd is open source software, is the technology of current comparative maturity, adopt logstash or fluentd technology, for the stability of system cloud gray model provides the foundation.
More preferably, in technique scheme, when described daily record memory scan device carries out data processing, that employing is elasticsearch.
The beneficial effect of above-mentioned further scheme is adopted to be: elasticsearch is open source software, it is the technology of current comparative maturity, data storage be adopt elasticsearch, ensure that accuracy during data processing, ensure that the stability that whole system is run.
Additionally provide a kind of offline logs and collect inquiry and analysis method, comprise the following steps:
Step S10: pusher obtains the daily record on node server, and with the form real time propelling movement of class discrimination on central server;
Step S20: supervising device is monitored the daily record composite catalog on central server, and the journal file in described daily record composite catalog is when changing, the path of the journal file changed and class label information are used in the message queue of storing data information as an information write;
Step S30: replay device reads the information comprising the log-file information changed from described message queue, and the content of the journal file changed is refitted in a unique file corresponding with the journal file classification changed;
Step S40: log collection cutter sweep reads the file after resetting from described replay device, carries out cutting, and store being sent to daily record memory scan device after the information unification data structure after cutting the file read;
Step S50: daily record memory scan device stores the information after the format received, and carries out category index;
Step S60: the api that data display device calls described daily record memory scan device retrieves, and the data retrieved is shown.
More preferably, in technique scheme, step S10 also comprises: described pusher carries out slicing treatment to the log information obtained from node server, and up-to-date section is pushed on central server.
Adopt the beneficial effect of above-mentioned further scheme to be: the slicing treatment that pusher carries out daily record, improve the efficiency of speed that log information pushes and problem investigation.
Accompanying drawing explanation
Below in conjunction with accompanying drawing, the invention will be further described:
Fig. 1 is the one-piece construction schematic diagram that offline logs of the present invention collects inquiry and analysis system;
Fig. 2 is the schematic diagram that offline logs of the present invention collects inquiry and analysis method.
Embodiment
Embodiment one
As shown in Figure 1, a kind of offline logs is provided to collect inquiry and analysis system, comprise pusher, supervising device, replay device, log collection cutter sweep, daily record memory scan device and data display device, pusher is for obtaining the daily record on node server, and with the form real time propelling movement of class discrimination on central server, supervising device is for monitoring the daily record composite catalog on central server, and the journal file in daily record composite catalog is when changing, the path of the journal file changed and class label information are used in the message queue of storing data information as an information write, replay device is used for from message queue, read the information comprising the log-file information changed, and the content of the journal file changed is refitted in a unique file corresponding with the journal file classification changed, log collection cutter sweep is used for reading the file after resetting from replay device, cutting is carried out to the file read, and the information after cutting is sent to daily record memory scan device with uniform data structure stores, daily record memory scan device is used for storing the information after the format received, and carry out category index, the api that data display device calls daily record memory scan device retrieves, and the data retrieved are shown.
As a kind of embodiment, pusher is multiple, is arranged on respectively on corresponding node server.Multiple pusher is arranged on corresponding node server, achieves the real time propelling movement of monitoring to numerous node server and log information, do not need to re-recognize for the server of login correspondence carry out inquiry log information, saved labour.
As a kind of embodiment, pusher carries out slicing treatment to the daily record on node server as required, and up-to-date section is sent to central server.To the slicing treatment that daily record on node server is carried out, improve the speed that log information pushes to a certain extent, improve the efficiency of problem investigation.
As a kind of embodiment, data structure is json structure after reunification.Adopt unified json structure to carry out the write of data, make the data of write more regular, and also more convenient when information reads, also more targeted.
As a kind of embodiment, the data message got is shown with the form of Web by data display device.The data message got is carried out Web displaying, facilitates the analysis and solve of problem types of technician to the position (server name or IP) of the node server of the problem of generation and generation.
As a kind of embodiment, supervising device is arranged on central server.Supervising device is arranged on central server, facilitates monitoring node server being sent to information on central server.
As a kind of embodiment, when log collection cutter sweep carries out data processing, that employing is logstash or fluentd.Logstash or fluentd is open source software, is the technology of current comparative maturity, adopts logstash or fluentd technology, for the stability of system cloud gray model provides the foundation.
As a kind of embodiment, when daily record memory scan device carries out data processing, that employing is elasticsearch.Elasticsearch is open source software, is the technology of current comparative maturity, is adopt elasticsearch, ensure that accuracy during data processing, ensure that the stability that whole system is run in data storage.
Embodiment two
As shown in Figure 2, on the basis of embodiment one, the offline logs provided collects inquiry and analysis method, comprises the following steps:
Step S10: pusher obtains the daily record on node server, and with the form real time propelling movement of class discrimination on central server;
Step S20: supervising device is monitored the daily record composite catalog on central server, and the journal file in daily record composite catalog is when changing, the path of the journal file changed and class label information are used in the message queue of storing data information as an information write;
Step S30: replay device reads the information comprising the log-file information changed from message queue, and the content of the journal file changed is refitted in a unique file corresponding with the journal file classification changed;
Step S40: log collection cutter sweep reads the file after resetting from replay device, carries out cutting, and the information after cutting is sent to daily record memory scan device with uniform data structure stores the file read;
Step S50: daily record memory scan device stores the information after the format received, and carries out category index;
Step S60: the api that data display device calls daily record memory scan device retrieves, and the data retrieved is shown.
As a kind of embodiment, step S10 also comprises: pusher carries out slicing treatment to the log information obtained from node server, and up-to-date section is pushed on central server.The slicing treatment that pusher carries out daily record, improves the speed of log information propelling movement and the efficiency of problem investigation.
The present invention is owing to adopting above technical scheme, its technique effect reached is: the present invention well achieves real-time collecting to a large amount of offline logs in zones of different, carries out unified log analysis and data exhibiting, effective by daily record centralized management, problem investigation and positioning problems are provided and operates easily, greatly save the time of issue handling, no longer need to log in corresponding server to check daily record and problem analysis, saved labour and production cost to a great extent.
Above-mentioned embodiment is intended to illustrate that the present invention can be professional and technical personnel in the field and realizes or use; modifying to above-mentioned embodiment will be apparent for those skilled in the art; therefore the present invention includes but be not limited to above-mentioned embodiment; any these claims or instructions of meeting describes; meet and principle disclosed herein and novelty, the method for inventive features, technique, product, all fall within protection scope of the present invention.
Claims (10)
1. offline logs collects an inquiry and analysis system, it is characterized in that: comprise pusher, supervising device, message queue, replay device, log collection cutter sweep, daily record memory scan device and data display device;
Described pusher for obtaining the daily record on node server, and with the form real time propelling movement of class discrimination on central server;
Described supervising device is for monitoring the daily record composite catalog on central server, and the journal file in described daily record composite catalog is when changing, the path of the journal file changed and class label information are used in the described message queue of storing data information as an information write;
Described replay device is used for from described message queue, read the information comprising the log-file information changed, and the content of the journal file changed is refitted in a unique file corresponding with the journal file classification changed;
Described log collection cutter sweep is used for reading the file after resetting from described replay device, carries out cutting, and the information after cutting is sent to described daily record memory scan device with uniform data structure stores the file read;
Described daily record memory scan device is used for storing the information after the format received, and carries out category index;
The api that described data display device calls described daily record memory scan device retrieves, and the data retrieved is shown.
2. offline logs as claimed in claim 1 collects inquiry and analysis system, it is characterized in that: described pusher is arranged on node server.
3. offline logs as claimed in claim 1 collects inquiry and analysis system, it is characterized in that: described pusher carries out slicing treatment to the daily record on node server as required, and up-to-date section is sent to central server.
4. offline logs as claimed in claim 1 collects inquiry and analysis system, it is characterized in that: data structure is json structure after reunification.
5. offline logs as claimed in claim 1 collects inquiry and analysis system, it is characterized in that: the data message got is shown with the form of Web by described data display device.
6. offline logs as claimed in claim 1 collects inquiry and analysis system, it is characterized in that: described supervising device is arranged on central server.
7. offline logs as claimed in claim 1 collects inquiry and analysis system, it is characterized in that: when described log collection cutter sweep carries out data processing, that employing is logstash or fluentd.
8. offline logs as claimed in claim 1 collects inquiry and analysis system, it is characterized in that: when described daily record memory scan device carries out data processing, that employing is elasticsearch.
9. offline logs collects an inquiry and analysis method, it is characterized in that: comprise the following steps:
Step S10: pusher obtains the daily record on node server, and with the form real time propelling movement of class discrimination on central server;
Step S20: supervising device is monitored the daily record composite catalog on central server, and the journal file in described daily record composite catalog is when changing, the path of the journal file changed and class label information are used in the message queue of storing data information as an information write;
Step S30: replay device reads the information comprising the log-file information changed from described message queue, and the content of the journal file changed is refitted in a unique file corresponding with the journal file classification changed;
Step S40: log collection cutter sweep reads the file after resetting from described replay device, carries out cutting, and the information after cutting is sent to daily record memory scan device with uniform data structure stores the file read;
Step S50: daily record memory scan device stores the information after the format received, and carries out category index;
Step S60: the api that data display device calls described daily record memory scan device retrieves, and the data retrieved is shown.
10. offline logs as claimed in claim 9 collects inquiry and analysis method, it is characterized in that: step S10 also comprises: described pusher carries out slicing treatment to the log information obtained from node server, and up-to-date section is pushed on central server.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510657081.7A CN105260452A (en) | 2015-10-12 | 2015-10-12 | System and method for collecting, searching, and analyzing offline log |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510657081.7A CN105260452A (en) | 2015-10-12 | 2015-10-12 | System and method for collecting, searching, and analyzing offline log |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105260452A true CN105260452A (en) | 2016-01-20 |
Family
ID=55100142
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510657081.7A Pending CN105260452A (en) | 2015-10-12 | 2015-10-12 | System and method for collecting, searching, and analyzing offline log |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105260452A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106844171A (en) * | 2016-12-27 | 2017-06-13 | 浪潮软件集团有限公司 | Mass operation and maintenance implementation method |
| CN108712329A (en) * | 2018-05-02 | 2018-10-26 | 山东汇贸电子口岸有限公司 | A kind of gateway and log recording retrieval device based on Elasticsearch |
| CN110262999A (en) * | 2019-06-03 | 2019-09-20 | 北京一览群智数据科技有限责任公司 | A kind of circulation of automated data and data processing method, shared file server |
| CN110263008A (en) * | 2019-06-20 | 2019-09-20 | 江苏满运软件科技有限公司 | Terminal offline logs management system, method, equipment and storage medium |
| CN110865986A (en) * | 2019-11-12 | 2020-03-06 | 深圳易为控股有限公司 | Data summarization synchronization scheme for enterprise retail system |
| CN111459755A (en) * | 2020-03-27 | 2020-07-28 | 海信视像科技股份有限公司 | Log file output control method and device based on EMMC life value |
| CN114168624A (en) * | 2021-12-08 | 2022-03-11 | 掌阅科技股份有限公司 | Data analysis method, computing device and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110191394A1 (en) * | 2010-01-29 | 2011-08-04 | Winteregg Joel | Method of processing log files in an information system, and log file processing system |
| CN102411533A (en) * | 2011-08-08 | 2012-04-11 | 浪潮电子信息产业股份有限公司 | Log-management optimizing method for clustered storage system |
| CN102750326A (en) * | 2012-05-30 | 2012-10-24 | 浪潮电子信息产业股份有限公司 | Log management optimization method of cluster system based on downsizing strategy |
| CN104036025A (en) * | 2014-06-27 | 2014-09-10 | 蓝盾信息安全技术有限公司 | Distribution-base mass log collection system |
| CN104714946A (en) * | 2013-12-11 | 2015-06-17 | 田鹏 | Large-scale Web log analysis system based on NoSQL |
-
2015
- 2015-10-12 CN CN201510657081.7A patent/CN105260452A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110191394A1 (en) * | 2010-01-29 | 2011-08-04 | Winteregg Joel | Method of processing log files in an information system, and log file processing system |
| CN102411533A (en) * | 2011-08-08 | 2012-04-11 | 浪潮电子信息产业股份有限公司 | Log-management optimizing method for clustered storage system |
| CN102750326A (en) * | 2012-05-30 | 2012-10-24 | 浪潮电子信息产业股份有限公司 | Log management optimization method of cluster system based on downsizing strategy |
| CN104714946A (en) * | 2013-12-11 | 2015-06-17 | 田鹏 | Large-scale Web log analysis system based on NoSQL |
| CN104036025A (en) * | 2014-06-27 | 2014-09-10 | 蓝盾信息安全技术有限公司 | Distribution-base mass log collection system |
Non-Patent Citations (1)
| Title |
|---|
| 马延辉等: "第8.1.2节数据类型", 《STORM企业级应用实战、运维和调优》 * |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106844171A (en) * | 2016-12-27 | 2017-06-13 | 浪潮软件集团有限公司 | Mass operation and maintenance implementation method |
| CN108712329A (en) * | 2018-05-02 | 2018-10-26 | 山东汇贸电子口岸有限公司 | A kind of gateway and log recording retrieval device based on Elasticsearch |
| CN110262999A (en) * | 2019-06-03 | 2019-09-20 | 北京一览群智数据科技有限责任公司 | A kind of circulation of automated data and data processing method, shared file server |
| CN110263008A (en) * | 2019-06-20 | 2019-09-20 | 江苏满运软件科技有限公司 | Terminal offline logs management system, method, equipment and storage medium |
| CN110263008B (en) * | 2019-06-20 | 2022-02-01 | 江苏满运软件科技有限公司 | Terminal off-line log management system, method, device and storage medium |
| CN110865986A (en) * | 2019-11-12 | 2020-03-06 | 深圳易为控股有限公司 | Data summarization synchronization scheme for enterprise retail system |
| CN110865986B (en) * | 2019-11-12 | 2023-05-12 | 深圳易为控股有限公司 | Enterprise retail system data summarization synchronization scheme |
| CN111459755A (en) * | 2020-03-27 | 2020-07-28 | 海信视像科技股份有限公司 | Log file output control method and device based on EMMC life value |
| CN114168624A (en) * | 2021-12-08 | 2022-03-11 | 掌阅科技股份有限公司 | Data analysis method, computing device and storage medium |
| CN114168624B (en) * | 2021-12-08 | 2022-09-20 | 掌阅科技股份有限公司 | Data analysis method, computing device and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105260452A (en) | System and method for collecting, searching, and analyzing offline log | |
| CN112612675B (en) | Distributed big data log link tracking method and system under micro-service architecture | |
| CN107943668B (en) | Computer server cluster log monitoring method and monitor supervision platform | |
| CN107133240B (en) | Page monitoring method, device and system | |
| CN109697456B (en) | Service analysis method, device, equipment and storage medium | |
| CN111209258A (en) | Tax end system log real-time analysis method, equipment, medium and system | |
| CN103412893A (en) | Collecting system and collecting method of logs | |
| CN110309130A (en) | A kind of method and device for host performance monitor | |
| CN103838867A (en) | Log processing method and device | |
| CN108268565B (en) | Method and system for processing user browsing behavior data based on data warehouse | |
| CN103425750A (en) | Cross-platform and cross-application log collecting system and collecting managing method thereof | |
| EP3432520B1 (en) | Efficient storage and querying of time series metrics | |
| CN108737170A (en) | A kind of batch daily record abnormal data alarm method and device | |
| CN104104734A (en) | Log analysis method and device | |
| CN105589791A (en) | Method for application system log monitoring management in cloud computing environment | |
| CN106021613A (en) | Bridge health monitoring system based on Hadoop | |
| CN111259073A (en) | Intelligent business system running state studying and judging system based on logs, flow and business access | |
| CN111614483A (en) | Link monitoring method and device, storage medium and computer equipment | |
| Osman et al. | Big data analytics and smart cities: A loose or tight couple? | |
| CN104462606A (en) | Method for determining diagnosis treatment measures based on log data | |
| CN109361576A (en) | A kind of PIM monitoring data processing method and system | |
| Agrawal et al. | Log-based cloud monitoring system for OpenStack | |
| CN113242159A (en) | Application access relation determining method and device | |
| CN108846072A (en) | A kind of data realization encapsulation technology based on visual analyzing | |
| CN108133043B (en) | Structured storage method for server running logs based on big data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20160120 |