CN109614553A - PaaS platform for log collection - Google Patents
PaaS platform for log collection Download PDFInfo
- Publication number
- CN109614553A CN109614553A CN201811573820.4A CN201811573820A CN109614553A CN 109614553 A CN109614553 A CN 109614553A CN 201811573820 A CN201811573820 A CN 201811573820A CN 109614553 A CN109614553 A CN 109614553A
- Authority
- CN
- China
- Prior art keywords
- mirror image
- log
- sleuth
- configuration
- elasticsearch
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/2866—Architectures; Arrangements
- H04L67/30—Profiles
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
Abstract
The invention relates to a kind of PaaS platforms for log collection.The platform include: include: at least one sleuth micro services end and ETK, in which: the ETK is connected to sleuth micro services end, for and being analyzed and processed to collected log by the acquisition log of sleuth micro services end.Unified filtering services are provided for platform provided by the present invention for the PaaS platform of log collection, to achieve the purpose that unified log.
Description
Technical field
The present invention relates to PaaS field of cloud computer technology, more particularly to a kind of PaaS platform for log collection.
Background technique
PaaS platform is (Platform-as-a-Service: platform services), operation and exploitation ring application service
Border is as a kind of business model that service provides.PaaS platform is capable of providing the middleware platform of being customized of enterprise research and development,
Cover database and application server etc. simultaneously.Wherein log collection is also the essential basic function component of PAAS platform.
Docker is the application container engine of an open source, and developer can be packaged their application and rely on packet and arrives
In one transplantable container, then it is published on the Linux machine of any prevalence, also may be implemented to virtualize.Container has been
Sandbox mechanism is entirely used, does not have any interface between each other.
PaaS platform is that multilingual is supported to develop, such as java, net, php etc..Since micro services are developed in platform
Language technology framework is not quite similar, and causes also be not quite similar to journal format.This can bring not tracking, the analysis of log
Just.
Summary of the invention
The technical problem to be solved in the present invention is to provide a kind of PaaS platforms for log collection, provide system for platform
One filtering services, to achieve the purpose that unified log.
In order to solve the above technical problems, the present invention provides the PaaS platform fast disposed of one kind, the platform include: to
A few sleuth micro services end and ETK, in which: the ETK is connected to sleuth micro services end, for by described
Sleuth micro services end acquires log, and is analyzed and processed to collected log.
As a kind of improvement of technical solution of the present invention, the ETK includes: logstash mirror image, is connected to described
Sleuth micro services end is responsible for being filtered the log at sleuth micro services end, collecting；Elasticsearch mirror image,
It is connected to the logstash mirror image, is responsible for scanning for the log that the logstash mirror image is collected, analyzing；And
Kibana mirror image is connected to the elasticsearch mirror image, is responsible for searching for the elasticsearch mirror image
Log is shown.
As a kind of improvement of technical solution of the present invention, the kibana mirror image is also responsible for the elasticsearch
The log that mirror image is searched for carries out visual analyzing.
As a kind of improvement of technical solution of the present invention, the logstash mirror image, the elasticsearch mirror image and
The kibana mirror image is the docker mirror image operated in docker container.
As a kind of improvement of technical solution of the present invention, the logstash mirror image, the elasticsearch mirror image and
The kibana mirror image is provided with the configuration file of itself.
As a kind of improvement of technical solution of the present invention, the configuration file of the logstash mirror image includes: responsible configuration
The output of the input configuration of log collection, the filter configuration for being responsible for configuration log filtering and the output of responsible configuration log
Configuration.
It include: end as a kind of improvement of technical solution of the present invention, in the configuration file of the elasticsearch mirror image
Mouth configuration and host configuration.
Include: as a kind of improvement of technical solution of the present invention, in the configuration file of the kibana mirror image port configuration,
Host configuration and the URL configuration of elasticsearch mirror image.
A kind of improvement as technical solution of the present invention, further includes: sleuth management end, the sleuth management end and every
A sleuth micro services end keeps network connection, controls for the operation to sleuth micro services end.
As a kind of improvement of technical solution of the present invention, sleuth micro services end and the sleuth management end are
Operate in the docker mirror image in docker container.
By adopting such a design, the present invention has at least the following advantages:
Unified filtering services are provided for platform, to achieve the purpose that unified log.
Detailed description of the invention
The above is merely an overview of the technical solutions of the present invention, in order to better understand the technical means of the present invention, below
In conjunction with attached drawing, the present invention is described in further detail with specific embodiment.
Fig. 1 is overall structure figure of the present invention for the PaaS platform of log collection.
Specific embodiment
Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings, it should be understood that preferred reality described herein
Apply example only for the purpose of illustrating and explaining the present invention and is not intended to limit the present invention.
The object of the present invention is to the log of PaaS platform standardization, unitized acquisition, solve PaaS platform and pass through log
Tracking is analyzed to obtain the dependence of micro services link as the monitoring function of PaaS platform and provide support.
Spring-cloud provides the component sleuth for being used to trace service.It can be taken by log
The dependence of business.Based on sleuth, the acquisition of distributed information log can be realized by existing logging tools.It is used herein
It is ELK, that is, elasticsearch, logstash, kibana.Sleuth,elasticsearch,logstash,
This four components of kibana are fabricated to docker image deployment by docker container technique and operate in PAAS platform cluster.
Logstash container mirror image: Logstash is an open source data collection engine, supports real time data function pipeline.
The data source that Logstash can be dispersed with Dynamic Integration, according to your selection to data normalization.According to different advanced downstreams
Analysis and visualization use-case comb data.
Elasticsearch container mirror image: Elasticsearch is the real-time search and analysis of a distributed expandable
Engine, a search engine established on the basis of full-text search engine Apache Lucene (TM).
Kibana container mirror image: Kibana be one open source analysis and Visualization Platform, designed for and
Elasticsearch works together.It is searched for, is checked with Kibana, and and the data that are stored in Elasticsearch index
It interacts.It can be easily carried out advanced data analysis, and visualize number in the form of various icons, table and map
According to.Kibana to understand that mass data becomes to be easy to.Its simple, interface based on browser enables you to quickly create
With shared dynamic instrument board, the variation of real-time display Elasticsearch inquiry.
Application service of the present invention uses Springcloud technological development, by the deployment operation of docker container.It is specific real
Apply that steps are as follows:
The first step, sleuth management end are realized:
Using springcloud web application quickly one web application service of exploitation, need to add in pom file
Sleuth is relied on.
Start service discovery is added in class note and the note of zipkin, starts this when and access the ground of the micro services
Location, it can be seen that the administration page of zipkin.Then jar packet is built by maven, using docker technology jar packet
Building is mirrored into.
Second step, the micro services end being managed are realized:
Simple configuration is needed at other micro services ends, brings the management of zipkin into.Pom file only needs to introduce
Rely on spring-cloud-sleuth-zipkin, spring-cloud-starter-sleuth, yml file addition configuration phase
The parameter configuration answered.Spring-cloud, logstash are to support logback, it is therefore desirable to configure phase for micro services
The logback-spring.xml answered recompilates building and is mirrored into.
Third step builds ELK system.
Elasticsearch is to be responsible for search and analysis log in ELK.In the configuration file of Elasticsearch
Port and host is configured, default port is 9200.
Logstash is to be responsible for collection and filtering log in ELK.The configuration file of Logstash must include three contents:
Input, filter and output.
Input: this module is responsible for collector journal, can read, read from redis or unlatching port allow production from file
The operation system of birthday will directly writes to logstash；Usage mode is in the present invention: unlatching port allows micro services directly to write
Enter.
Filter: this module is responsible for the log being collected by filtration, and according to after filtering to log definition display field.
Output: this module is responsible for filtered log being output to elasticsearch or file, redis etc..
Usage mode is in the present invention: filtered log is output to elasticsearch.
Kibana is to be responsible for showing log in ELK.In the configuration file of kibana configure port, host and
elasticsearch url.Kibana default port is 5601；Elasticsearch url format is http://ip:port.
The above described is only a preferred embodiment of the present invention, be not intended to limit the present invention in any form, this
Field technical staff makes a little simple modification, equivalent variations or modification using the technology contents of the disclosure above, all falls within this hair
In bright protection scope.
Claims (10)
1. a kind of PaaS platform for log collection characterized by comprising at least one sleuth micro services end and ETK,
Wherein:
The ETK is connected to sleuth micro services end, for acquiring log by sleuth micro services end, and it is right
Collected log is analyzed and processed.
2. the PaaS platform according to claim 1 for log collection, which is characterized in that the ETK includes:
Logstash mirror image is connected to sleuth micro services end, is responsible for the log to sleuth micro services end and carries out
Filtering is collected；
Elasticsearch mirror image is connected to the logstash mirror image, is responsible for the log collected to the logstash mirror image
It scans for, analyze；And
Kibana mirror image is connected to the elasticsearch mirror image, is responsible for searching for the elasticsearch mirror image
To log be shown.
3. the PaaS platform according to claim 2 for log collection, which is characterized in that the kibana mirror image is also negative
It blames and visual analyzing is carried out to the log that the elasticsearch mirror image is searched for.
4. the PaaS platform according to claim 2 for log collection, which is characterized in that the logstash mirror image,
The elasticsearch mirror image and the kibana mirror image are the docker mirror images operated in docker container.
5. the PaaS platform according to claim 2 for log collection, which is characterized in that the logstash mirror image,
The elasticsearch mirror image and the kibana mirror image are provided with the configuration file of itself.
6. the PaaS platform according to claim 5 for log collection, which is characterized in that the logstash mirror image
Configuration file includes: the input configuration of responsible configuration log collection, is responsible for the filter configuration of configuration log filtering, and is born
Blame the output configuration of configuration log output.
7. the PaaS platform according to claim 5 for log collection, which is characterized in that the elasticsearch
It include: port configuration and host configuration in the configuration file of mirror image.
8. the PaaS platform according to claim 5 for log collection, which is characterized in that the kibana mirror image is matched
Setting includes: port configuration in file, host configures and the URL of elasticsearch mirror image configuration.
9. the PaaS platform according to claim 1 for log collection, which is characterized in that further include:
Sleuth management end, the sleuth management end and each sleuth micro services end keep being connected to the network, for pair
The operation at sleuth micro services end is controlled.
10. the PaaS platform according to claim 9 for log collection, which is characterized in that the sleuth micro services
End and the sleuth management end are the docker mirror images operated in docker container.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811573820.4A CN109614553A (en) | 2018-12-21 | 2018-12-21 | PaaS platform for log collection |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811573820.4A CN109614553A (en) | 2018-12-21 | 2018-12-21 | PaaS platform for log collection |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109614553A true CN109614553A (en) | 2019-04-12 |
Family
ID=66010346
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811573820.4A Pending CN109614553A (en) | 2018-12-21 | 2018-12-21 | PaaS platform for log collection |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109614553A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111008112A (en) * | 2019-12-17 | 2020-04-14 | 紫光云(南京)数字技术有限公司 | Elk centralized log system built based on docker |
| WO2021017614A1 (en) * | 2019-07-31 | 2021-02-04 | 平安科技（深圳）有限公司 | Threat intelligence data collection and processing method and system, apparatus, and storage medium |
| CN113282557A (en) * | 2021-04-27 | 2021-08-20 | 联通(江苏)产业互联网有限公司 | Big data log analysis method and system based on Spring framework |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160269482A1 (en) * | 2015-03-12 | 2016-09-15 | International Business Machines Corporation | Providing agentless application performance monitoring (apm) to tenant applications by leveraging software-defined networking (sdn) |
| CN106649044A (en) * | 2016-12-28 | 2017-05-10 | 深圳市深信服电子科技有限公司 | Log processing method, device and system based on container cloud system |
| CN107682351A (en) * | 2017-10-20 | 2018-02-09 | 携程旅游网络技术（上海）有限公司 | Method, system, equipment and the storage medium of network security monitoring |
| CN107690623A (en) * | 2015-05-28 | 2018-02-13 | 甲骨文国际公司 | Automatic abnormality detection and solution system |
| CN108376181A (en) * | 2018-04-24 | 2018-08-07 | 丹阳飓风物流股份有限公司 | Log services platform based on ELK |
| CN108446111A (en) * | 2018-03-26 | 2018-08-24 | 国家电网公司客户服务中心 | A kind of micro services construction method based on Spring cloud |
-
2018
- 2018-12-21 CN CN201811573820.4A patent/CN109614553A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160269482A1 (en) * | 2015-03-12 | 2016-09-15 | International Business Machines Corporation | Providing agentless application performance monitoring (apm) to tenant applications by leveraging software-defined networking (sdn) |
| CN107690623A (en) * | 2015-05-28 | 2018-02-13 | 甲骨文国际公司 | Automatic abnormality detection and solution system |
| CN106649044A (en) * | 2016-12-28 | 2017-05-10 | 深圳市深信服电子科技有限公司 | Log processing method, device and system based on container cloud system |
| CN107682351A (en) * | 2017-10-20 | 2018-02-09 | 携程旅游网络技术（上海）有限公司 | Method, system, equipment and the storage medium of network security monitoring |
| CN108446111A (en) * | 2018-03-26 | 2018-08-24 | 国家电网公司客户服务中心 | A kind of micro services construction method based on Spring cloud |
| CN108376181A (en) * | 2018-04-24 | 2018-08-07 | 丹阳飓风物流股份有限公司 | Log services platform based on ELK |
Non-Patent Citations (3)
| Title |
|---|
| 严丽云 等: "容器技术及可行应用场景分析", 《广东通信技术》, pages 191 - 199 * |
| 董长青等: "基于封装和集成技术的工业APP应用研究", 《工程机械》 * |
| 董长青等: "基于封装和集成技术的工业APP应用研究", 《工程机械》, vol. 49, no. 12, 10 December 2018 (2018-12-10), pages 2 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021017614A1 (en) * | 2019-07-31 | 2021-02-04 | 平安科技（深圳）有限公司 | Threat intelligence data collection and processing method and system, apparatus, and storage medium |
| CN111008112A (en) * | 2019-12-17 | 2020-04-14 | 紫光云(南京)数字技术有限公司 | Elk centralized log system built based on docker |
| CN113282557A (en) * | 2021-04-27 | 2021-08-20 | 联通(江苏)产业互联网有限公司 | Big data log analysis method and system based on Spring framework |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP2932418B1 (en) | Metadata driven real-time analytics framework | |
| US11537552B2 (en) | Rule generation in a data governance framework | |
| Trakadas et al. | Comparison of management and orchestration solutions for the 5G era | |
| US9785431B2 (en) | Development, test and deployment of applications | |
| CN109614553A (en) | PaaS platform for log collection | |
| Serrano et al. | Defining the stack for service delivery models and interoperability in the Internet of Things: A practical case with OpenIoT-VDK | |
| CN109684370A (en) | Daily record data processing method, system, equipment and storage medium | |
| US20170053242A1 (en) | System and Method for a Big Data Analytics Enterprise Framework | |
| CN101917286B (en) | Web performance analysis system and method for realizing cross-browser in internet application system | |
| EP2463810A1 (en) | Data extraction framework | |
| CN108964968A (en) | Service access management method and system under a kind of container cloud environment | |
| US11361039B2 (en) | Autodidactic phenological data collection and verification | |
| CN108509254A (en) | The monitoring method and system of Docker Container | |
| US20140237554A1 (en) | Unified platform for big data processing | |
| CN110413586A (en) | Distributed information log management method and system | |
| JP2019510282A (en) | Generating a streaming analysis application using glossary | |
| CN113268464B (en) | Log display method and device, electronic equipment and storage medium | |
| CN106776314A (en) | A kind of test system | |
| US10776359B2 (en) | Abstractly implemented data analysis systems and methods therefor | |
| CN110019044A (en) | Big data cluster quasi real time Yarn Mission Monitor analysis method | |
| WO2020123692A2 (en) | Internet of things system topology generation | |
| US10956026B2 (en) | Smart element filtering method via gestures | |
| US20240086239A1 (en) | Services development and deployment for backend system integration | |
| CN109446648A (en) | Service of simulation method for building up and device | |
| US9767170B2 (en) | Storage area network zone optimization |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |