CN108376181A - Log services platform based on ELK - Google Patents
Log services platform based on ELK Download PDFInfo
- Publication number
- CN108376181A CN108376181A CN201810373492.7A CN201810373492A CN108376181A CN 108376181 A CN108376181 A CN 108376181A CN 201810373492 A CN201810373492 A CN 201810373492A CN 108376181 A CN108376181 A CN 108376181A
- Authority
- CN
- China
- Prior art keywords
- modules
- logstash
- cluster
- daily record
- elasticsearch
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/1734—Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/08—Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/55—Push-based network services
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/16—Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP]
- H04L69/164—Adaptation or special uses of UDP protocol
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Business, Economics & Management (AREA)
- Computational Linguistics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Economics (AREA)
- Development Economics (AREA)
- Operations Research (AREA)
- Human Resources & Organizations (AREA)
- General Business, Economics & Management (AREA)
- Entrepreneurship & Innovation (AREA)
- Quality & Reliability (AREA)
- Tourism & Hospitality (AREA)
- Computer Security & Cryptography (AREA)
- Strategic Management (AREA)
- Marketing (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Software Systems (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses the log services platforms based on ELK, including：Logstash modules, the daily record file-based generated for collection system；Redis cluster modules, for receiving collected by Logstash modules and caching the daily record pushed in real time into memory and by UDP；Logstash cluster modules, the daily record for receiving the output of Redis cluster modules；Load equalizer, with Logstash cluster module data associations；ElasticSearch cluster modules, the daily record for the output of Logstash cluster modules to be written；Kibana modules are connected with ElasticSearch cluster modules；The log audit module being connected with the ElasticSearch databases in ElasticSearch cluster modules；SDK for analyzing the data-interface response time, SDK include overtime warning module, and above-mentioned platform provides all multimodes and is convenient for recording the daily record of different storage modes, is convenient for trouble-shoots, audit analysis and performance evaluation.
Description
Technical field
The present invention relates to logistics fields, more particularly to the log services platform based on ELK.
Background technology
Nowadays, because our company is quickly grown, the function of original logistics management related system is more and more, meanwhile, also increase
Many other operation systems, thus lead to some systems when something goes wrong, it cannot timely and effectively find, cause to influence work
Quality, therefore, our company determine to research and develop special log service system.
In conclusion how to ensure that whole systems can work normally, and can have in time when going wrong
Pinpointing the problems for effect, becomes those skilled in the art's technical problem urgently to be resolved hurrily.
Invention content
The object of the present invention is to provide the log services platform based on ELK, it can ensure that whole systems can work normally,
And it can timely and effectively pinpoint the problems when going wrong.
In order to solve the above technical problems, the present invention provides following technical solutions：
Log services platform based on ELK, including：Logstash modules generate file-based for collection system
Daily record；
Redis cluster modules, for receiving collected by Logstash modules and caching into memory and pass through in real time
The daily record that UDP is pushed；
Logstash cluster modules, the daily record for receiving the output of Redis cluster modules；
Load equalizer, with Logstash cluster module data associations；ElasticSearch cluster modules, for being written
The daily record of Logstash cluster modules output；
Kibana modules are connected with ElasticSearch cluster modules；
The log audit module being connected with the ElasticSearch databases in ElasticSearch cluster modules；
SDK for analyzing the data-interface response time, SDK include overtime warning module.
Log services platform provided by the present invention based on ELK is for all operation systems and other dependence systems
The log recording platform of system provides all multimodes and is convenient for recording the daily record of different storage modes, convenient for trouble-shoots, audit point
Analysis and performance evaluation.
Description of the drawings
In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, to embodiment or will show below
There is attached drawing needed in technology description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this
Some embodiments of invention for those of ordinary skill in the art without creative efforts, can be with
Obtain other attached drawings according to these attached drawings.
Fig. 1 is the schematic diagram of the log services platform provided by the present invention based on ELK.
Specific implementation mode
Core of the invention is to provide a kind of log services platform based on ELK, can ensure that whole systems can be normal
Work, and can timely and effectively pinpoint the problems when going wrong.
Following will be combined with the drawings in the embodiments of the present invention, and technical solution in the embodiment of the present invention carries out clear, complete
Site preparation describes, it is clear that described embodiments are only a part of the embodiments of the present invention, instead of all the embodiments.It is based on
Embodiment in the present invention, it is obtained by those of ordinary skill in the art without making creative efforts every other
Embodiment shall fall within the protection scope of the present invention.
Incorporated by reference to Fig. 1, Fig. 1 is the schematic diagram of the log services platform provided by the present invention based on ELK.
Log services platform provided by the present invention based on ELK, including：Logstash modules are produced for collection system
Raw daily record file-based；Redis cluster modules, for receiving collected by Logstash modules and caching into memory
And the daily record pushed in real time by UDP；Logstash cluster modules, the daily record for receiving the output of Redis cluster modules；It is negative
Balanced device is carried, with Logstash cluster module data associations；ElasticSearch cluster modules, for Logstash collection to be written
The daily record of group's module output；Kibana modules are connected with ElasticSearch cluster modules；With ElasticSearch cluster moulds
The connected log audit module of ElasticSearch databases in the block；SDK for analyzing the data-interface response time, SDK
Including overtime warning module.
Logstash modules are responsible for collector journal, and because of the particularity of our systems, we pass through on specific virtual machine
The daily record file-based that system generates is collected and then is pushed to special Logstash clusters by Logstash modules, and
The daily record generated in our systems, log buffer is then pushed into memory and by UDP by the way of built-in by we in real time
Logstash clusters, and externally only there are one public IP address for Logstash clusters, because we use one on the cluster
Layer load balancing, occur in Logstash clusters in this way any one node it is unavailable after, load equalizer can be found rapidly
And reject it from enabled node, certain post-service can be rejoined in enabled node if having restored.
Increase a Redis cluster be responsible for buffer daily record, can ensure so no matter current log peak value it is how high, day
Aspiration system can stable operation, ensure that each daily record can record, will not be interrupted suddenly because of operation system leads to its oneself
The daily record of oneself caching, which does not push also, just to disappear.
ElasticSearch be belong to full-text index database, as long as so index field can be according to any value
It is inquired, can be matched a letter in only one big section exception word of inquiry and quick search arrives, inquired
It is wider compared to support range with similar product in terms of sentence support.
In addition to above-mentioned general query, required data can also be inquired according to query statement then according to certain field
It carries out aggregate query and generates various types of pictorial statements, without the intervention of any development cost, to reach real-time prison
Control, such as abnormal number, the simple pictorial statement such as number of request and order volume.
Based on the front-end interface of the audit of ElasticSearch database developments one, log audit module can directly from
Record analysis and the unloading that our needs are inquired in a large amount of daily record data, do not have any cost for operation system, by
In the integrality of log recording, what final audit log can be very perfect records the behavior operated, data and as a result, also can
Enough support the inquiry, such as employee, time and order etc. of various dimensions.
All platforms are all intercepted and have been recorded by the way of own SDK, can be according to its corresponding time analysis
The response time for going out total interface not only can simply check the response time in different time periods the interface of emphasis,
Early warning mechanism is also added simultaneously, for example the request average time of special interface is more than more than devaluation or longest request time
Specifically devalue that we can record and be notified by way of mail or short message corresponding operation maintenance personnel, ensures system energy
Enough efficient work, because of the case where many times system itself does not occur delay machine, and is only merely that interim peak value is led
The response speed of cause system is slack-off, and at this time by Performance Analysis Platform, we can be with quick response, and starts increase machine in advance
System still can efficiently work when device node guarantee peak value.
A kind of log services platform based on ELK provided by the present invention is described in detail above.It answers herein
With specific case, principle and implementation of the present invention are described, and the explanation of above example is only intended to help to manage
Solve the method and its core concept of the present invention.It should be pointed out that for those skilled in the art, not departing from
, can be with several improvements and modifications are made to the present invention under the premise of the principle of the invention, these improvement and modification also fall into this hair
In bright scope of the claims.
Claims (1)
1. the log services platform based on ELK, which is characterized in that including：
Logstash modules, the daily record file-based generated for collection system；
Redis cluster modules, for receiving collected by Logstash modules and caching into memory and in real time by UDP institutes
The daily record of push；
Logstash cluster modules, the daily record for receiving the output of Redis cluster modules；
Load equalizer, with Logstash cluster module data associations；
ElasticSearch cluster modules, the daily record for the output of Logstash cluster modules to be written；
Kibana modules are connected with ElasticSearch cluster modules；
The log audit module being connected with the ElasticSearch databases in ElasticSearch cluster modules；
SDK for analyzing the data-interface response time, the SDK include overtime warning module.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810373492.7A CN108376181A (en) | 2018-04-24 | 2018-04-24 | Log services platform based on ELK |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810373492.7A CN108376181A (en) | 2018-04-24 | 2018-04-24 | Log services platform based on ELK |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108376181A true CN108376181A (en) | 2018-08-07 |
Family
ID=63032731
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810373492.7A Pending CN108376181A (en) | 2018-04-24 | 2018-04-24 | Log services platform based on ELK |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108376181A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109614553A (en) * | 2018-12-21 | 2019-04-12 | 北京博明信德科技有限公司 | PaaS platform for log collection |
| CN109840251A (en) * | 2018-12-29 | 2019-06-04 | 北京奥鹏远程教育中心有限公司 | A kind of big data aggregate query method |
| CN110097324A (en) * | 2019-05-14 | 2019-08-06 | 北京顺丰同城科技有限公司 | A kind of logistics information querying method and device |
| CN109492037B (en) * | 2018-11-02 | 2020-06-12 | 北京明朝万达科技股份有限公司 | Data acquisition method and device based on Redis and Logstash |
| CN111309793A (en) * | 2020-01-15 | 2020-06-19 | 北大方正集团有限公司 | Data processing method, device and equipment |
| CN112328568A (en) * | 2020-11-03 | 2021-02-05 | 杭州天宽科技有限公司 | Service system log recording method and system based on SDK (software development kit) packet |
| CN111858274B (en) * | 2020-07-02 | 2021-06-01 | 北京睿知图远科技有限公司 | Stability monitoring method for big data scoring system |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103399887A (en) * | 2013-07-19 | 2013-11-20 | 蓝盾信息安全技术股份有限公司 | Query and statistical analysis system for mass logs |
| CN106130782A (en) * | 2016-07-19 | 2016-11-16 | 努比亚技术有限公司 | A kind of method and system obtaining server log |
| CN107861859A (en) * | 2017-11-22 | 2018-03-30 | 北京汇通金财信息科技有限公司 | A kind of blog management method and system based on micro services framework |
-
2018
- 2018-04-24 CN CN201810373492.7A patent/CN108376181A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103399887A (en) * | 2013-07-19 | 2013-11-20 | 蓝盾信息安全技术股份有限公司 | Query and statistical analysis system for mass logs |
| CN106130782A (en) * | 2016-07-19 | 2016-11-16 | 努比亚技术有限公司 | A kind of method and system obtaining server log |
| CN107861859A (en) * | 2017-11-22 | 2018-03-30 | 北京汇通金财信息科技有限公司 | A kind of blog management method and system based on micro services framework |
Non-Patent Citations (2)
| Title |
|---|
| 刘必雄等: "基于集群的多源日志综合审计系统", 《计算机应用》 * |
| 郑清泉: "基于云计算技术的通用日志管理构件", 《广东通信技术》 * |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109492037B (en) * | 2018-11-02 | 2020-06-12 | 北京明朝万达科技股份有限公司 | Data acquisition method and device based on Redis and Logstash |
| CN109614553A (en) * | 2018-12-21 | 2019-04-12 | 北京博明信德科技有限公司 | PaaS platform for log collection |
| CN109840251A (en) * | 2018-12-29 | 2019-06-04 | 北京奥鹏远程教育中心有限公司 | A kind of big data aggregate query method |
| CN109840251B (en) * | 2018-12-29 | 2023-11-03 | 北京奥鹏远程教育中心有限公司 | Big data aggregation query method |
| CN110097324A (en) * | 2019-05-14 | 2019-08-06 | 北京顺丰同城科技有限公司 | A kind of logistics information querying method and device |
| CN111309793A (en) * | 2020-01-15 | 2020-06-19 | 北大方正集团有限公司 | Data processing method, device and equipment |
| CN111858274B (en) * | 2020-07-02 | 2021-06-01 | 北京睿知图远科技有限公司 | Stability monitoring method for big data scoring system |
| CN112328568A (en) * | 2020-11-03 | 2021-02-05 | 杭州天宽科技有限公司 | Service system log recording method and system based on SDK (software development kit) packet |
| CN112328568B (en) * | 2020-11-03 | 2023-06-02 | 杭州天宽科技有限公司 | Service system log recording method and system based on SDK package |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108376181A (en) | Log services platform based on ELK | |
| US20180365085A1 (en) | Method and apparatus for monitoring client applications | |
| CN109885453B (en) | Big data platform monitoring system based on stream data processing | |
| CN109871392B (en) | Slow sql real-time data acquisition method under distributed application system | |
| CN107229556A (en) | Log Analysis System based on elastic components | |
| US7562139B2 (en) | System and method for logging events of network devices | |
| CN103475535A (en) | Log management system of cloud computing server | |
| CN112199394A (en) | Alarm information pushing method and system, intelligent terminal and storage medium | |
| CN106095575B (en) | A kind of devices, systems, and methods of log audit | |
| CN110209518A (en) | A kind of multi-data source daily record data, which is concentrated, collects storage method and device | |
| US20200372372A1 (en) | Predicting the disaster recovery invocation response time | |
| CN111314158B (en) | Big data platform monitoring method, device, equipment and medium | |
| CN113783931A (en) | Internet of things data aggregation and analysis method | |
| CN109933572A (en) | A kind of data managing method and system for large enterprise | |
| CN112965979A (en) | User behavior analysis method and device and electronic equipment | |
| CN110908815A (en) | Message queue data early warning method, device and system and storage medium | |
| CN110555452A (en) | network problem processing method and device based on intelligent clustering | |
| CN115038083A (en) | Telecom fraud early warning identification method and system applied to AI operator industry | |
| CN107257289A (en) | A kind of risk analysis equipment, monitoring system and monitoring method | |
| CN108173711B (en) | Data exchange monitoring method for internal system of enterprise | |
| CN112465480A (en) | A real name system management system of labor affairs for building trade | |
| CN112579552A (en) | Log storage and calling method, device and system | |
| CN112346938B (en) | Operation auditing method and device, server and computer readable storage medium | |
| CN117131080A (en) | Data processing platform based on stream processing and message queue | |
| CN112579391A (en) | Distributed database automatic operation and maintenance method and system based on artificial intelligence |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20180807 |