CN104113431A - Distributed log acquisition pipeline design acting on cloud computing data center - Google Patents
Distributed log acquisition pipeline design acting on cloud computing data center Download PDFInfo
- Publication number
- CN104113431A CN104113431A CN201410056760.4A CN201410056760A CN104113431A CN 104113431 A CN104113431 A CN 104113431A CN 201410056760 A CN201410056760 A CN 201410056760A CN 104113431 A CN104113431 A CN 104113431A
- Authority
- CN
- China
- Prior art keywords
- log
- daily record
- collection
- cloud computing
- log acquisition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Debugging And Monitoring (AREA)
Abstract
The invention relates to a pipeline design, and particularly relates to a distributed log acquisition pipeline design acting on a cloud computing data center. The technical scheme is that the system is composed of multiple log acquisition instances. Multiple log acquisition plug-ins are operated in each log acquisition instance. Log acquisition in each instance acts as one process of an operating system to operate. Log parameters on different network devices are configured according to user requirements. Log information is sent to the log acquisition instances, and the log data information is stored in a log storage cluster in a concentrated way by the log acquisition instances. The beneficial effects are that a contradiction between diversity in modes of network device log acquisition in cloud computing and unification in the modes of log receiving of a storage server in the log cluster is solved.
Description
Technical field
The present invention relates to a kind of pipe design, be specifically related to a kind of distributed information log that acts on cloud computing data center and gather pipe design.
Background technology
Along with the development of cloud computing technology, in cloud computing, provide the network equipment of various functions more and more, be accompanied by the increase of various network device, variety classes, dissimilar, also corresponding increasing of the daily record of same-action not.In cloud computing, the importance of log management is self-evident, and log collection is wherein that log management is basic.Contradiction in the face of the mode variation of network equipment log collection in cloud computing and the unification of daily record centralized stores server reception daily record mode, this patent gathers pipeline by a kind of distributed information log of design, solve this contradiction, between a plurality of Log Sources and one or more daily record centralized stores server, set up adaptive pipeline.
Summary of the invention
A kind of distributed information log that acts on cloud computing data center of this patent invention gathers pipe design, code name is LogPipe, this system is comprised of a plurality of log collection examples, in each log collection example, move a plurality of log collection plug-in units, in each example, log collection is as a process operation of operating system, according to user's demand, daily record parameter on configuration heterogeneous networks equipment, these log informations are sent on log collection example, log collection example is concentrated daily record data information to store in log store cluster, and the Log Source network equipment and log store server do not need to make any modification, to the collection of Log Source adaptive with the output adaptation of log store server by distributed, the log collection pipeline of clustering is done.
1. log information gathers example
LogPipe is deployed on the network equipment that need to gather daily record, the active of daily record input module or the various daily record raw information of passive reception Log Source by LogPipe, then the processing such as (necessary words) of transferring to the log processing module of LogPipe to filter, the daily record output module by LogPipe after handling outputs to one or more log store servers the daily record of processing.
The design of LogPipe input module, filtering module and output module all adopts card format, can accomplish like this to use same pipeline, adapts to different situations because the plug-in unit of the each several part using is different.
In order to achieve the above object, the present invention proposes a kind of distributed information log that acts on cloud computing data center and gather pipe design, comprise following part:
1) log collection, daily record input module is to dissimilar, and the daily record of agreement gathers.
2) daily record is filtered, and the daily record collecting is filtered etc. to processing.
3) daily record output, outputs to log server to the daily record of processing.
2. log collection plug-in unit
Log collection plug-in component operation is in each log collection example, pass through Plugin Mechanism, log collection example can gather dissimilar log information, for example, standard logs information Syslog, WMI, SNMP Trap, file type daily records etc., process by Log Types the autgmentability that plug-in unitization can improve system easily, improve the compatibility of system.
3. log collection cluster
Log collection cluster is the distributed treatment mechanism adopting for a large amount of log informations of cloud computing data center, each log collection example can gather the log information in certain limit, the processing by log collection example, forward the log information after processing can be sent to higher level's log collection example or at a high speed, in Large Volume Data buffer memory.So, by many levels, the log collection of clustering is disposed, and can tackle a large amount of log informations of generation cloud computing data center in.
The present invention designs a kind of distributed information log that acts on cloud computing data center and gathers pipe design, can solve the contradiction of mode variation with the unification of daily record centralized stores server reception daily record mode of network equipment log collection in cloud computing by this system.
accompanying drawing explanation:
Fig. 1 is network design schematic diagram of the present invention;
Fig. 2 internal structure schematic diagram of the present invention;
Fig. 3 daily record flow processing schematic diagram.
Embodiment
A kind of distributed information log that acts on cloud computing data center proposing for the clearer the present invention of introduction gathers pipe design, below in conjunction with specific embodiment, is further described.
This patent invention clustering log collection pipeline code name LogPipe, this system is comprised of a plurality of log collection examples, in each log collection example, move a plurality of log collection plug-in units, in each example, log collection is as a process operation of operating system, according to user's demand, daily record parameter on configuration heterogeneous networks equipment, these log informations are sent on log collection example, log collection example is concentrated daily record data information to store in log store cluster, and the Log Source network equipment and log store server do not need to make any modification, to the collection of Log Source adaptive with the output adaptation of log store server by distributed, the log collection pipeline of clustering is done.
By reference to the accompanying drawings 1, LogPipe is deployed on the Log Source that needs to gather, and gathers daily record be forwarded on log store server by LogPipe.
In conjunction with Fig. 2, LogPipe internal structure is described in detail, mainly comprise with lower module:
The daily record input module of LogPipe is responsible for initiatively or passive receive daily record.
The log processing module of LogPipe is responsible for filtering daily record.
The daily record output module of LogPipe is responsible for output journal to one or more log servers.
In conjunction with Fig. 3, daily record flow processing detailed process is described:
(1) by the daily record input module of LogPipe initiatively or the various daily record raw information of passive reception Log Source;
(2) log processing module by LogPipe filters processing such as (necessary words);
(3) the daily record output module by LogPipe outputs to one or more log store servers the daily record of processing.
The above is only the preferred embodiment of the present invention; it should be pointed out that for those skilled in the art, do not departing under the prerequisite of the technology of the present invention principle; can also make some improvements and modifications, these improvements and modifications also should be considered as protection scope of the present invention.
Claims (1)
1. a distributed information log that acts on cloud computing data center gathers pipe design, the pipeline that serves as Log Source and log store server, the mode variation that realizes log collection is docked with the good of unification that daily record centralized stores server receives daily record mode, it is characterized in that, its technical scheme is that system is comprised of a plurality of log collection examples, in each log collection example, move a plurality of log collection plug-in units, in each example, log collection is as a process operation of operating system, according to user's demand, daily record parameter on configuration heterogeneous networks equipment, these log informations are sent on log collection example, log collection example is concentrated daily record data information to store in log store cluster, and the Log Source network equipment and log store server do not need to make any modification, to the collection of Log Source adaptive with the output adaptation of log store server by distributed, the log collection pipeline of clustering is done.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410056760.4A CN104113431A (en) | 2014-02-20 | 2014-02-20 | Distributed log acquisition pipeline design acting on cloud computing data center |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410056760.4A CN104113431A (en) | 2014-02-20 | 2014-02-20 | Distributed log acquisition pipeline design acting on cloud computing data center |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN104113431A true CN104113431A (en) | 2014-10-22 |
Family
ID=51710068
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410056760.4A Pending CN104113431A (en) | 2014-02-20 | 2014-02-20 | Distributed log acquisition pipeline design acting on cloud computing data center |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104113431A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104967696A (en) * | 2015-07-29 | 2015-10-07 | 网神信息技术（北京）股份有限公司 | Log sending method and device |
| CN106406834A (en) * | 2015-07-30 | 2017-02-15 | 中兴通讯股份有限公司 | Method, device and system for managing application system |
| FR3042934A1 (en) * | 2015-10-23 | 2017-04-28 | Rizze | SYSTEM AND AUTOMATIC METHOD FOR CONCENTRATING APPLICATION TRACES IN A CLOUD OPERATING SYSTEM COMPRISING ONE OR MORE KNOTS |
| CN107864056A (en) * | 2017-11-04 | 2018-03-30 | 公安部第三研究所 | A kind of distributed event acquisition probe, distributed event high speed acquisition system and method |
| CN113179302A (en) * | 2021-04-19 | 2021-07-27 | 杭州海康威视系统技术有限公司 | Log system, and method and device for collecting log data |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7743030B1 (en) * | 2006-09-29 | 2010-06-22 | Emc Corporation | Methods and apparatus for declarative log collection |
| CN102307111A (en) * | 2011-09-02 | 2012-01-04 | 深圳中兴网信科技有限公司 | Log distribution acquisition analysis method and system thereof |
| CN103412893A (en) * | 2013-07-24 | 2013-11-27 | 广东电子工业研究院有限公司 | Collecting system and collecting method of logs |
| CN103425750A (en) * | 2013-07-23 | 2013-12-04 | 国云科技股份有限公司 | Cross-platform and cross-application log collecting system and collecting managing method thereof |
-
2014
- 2014-02-20 CN CN201410056760.4A patent/CN104113431A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7743030B1 (en) * | 2006-09-29 | 2010-06-22 | Emc Corporation | Methods and apparatus for declarative log collection |
| CN102307111A (en) * | 2011-09-02 | 2012-01-04 | 深圳中兴网信科技有限公司 | Log distribution acquisition analysis method and system thereof |
| CN103425750A (en) * | 2013-07-23 | 2013-12-04 | 国云科技股份有限公司 | Cross-platform and cross-application log collecting system and collecting managing method thereof |
| CN103412893A (en) * | 2013-07-24 | 2013-11-27 | 广东电子工业研究院有限公司 | Collecting system and collecting method of logs |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104967696A (en) * | 2015-07-29 | 2015-10-07 | 网神信息技术（北京）股份有限公司 | Log sending method and device |
| CN106406834A (en) * | 2015-07-30 | 2017-02-15 | 中兴通讯股份有限公司 | Method, device and system for managing application system |
| FR3042934A1 (en) * | 2015-10-23 | 2017-04-28 | Rizze | SYSTEM AND AUTOMATIC METHOD FOR CONCENTRATING APPLICATION TRACES IN A CLOUD OPERATING SYSTEM COMPRISING ONE OR MORE KNOTS |
| CN107864056A (en) * | 2017-11-04 | 2018-03-30 | 公安部第三研究所 | A kind of distributed event acquisition probe, distributed event high speed acquisition system and method |
| CN113179302A (en) * | 2021-04-19 | 2021-07-27 | 杭州海康威视系统技术有限公司 | Log system, and method and device for collecting log data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104113431A (en) | Distributed log acquisition pipeline design acting on cloud computing data center | |
| EP2753024A3 (en) | System and method for continuously monitoring and searching social networking media | |
| CN103942210A (en) | Processing method, device and system of mass log information | |
| CN102594598A (en) | Log management system and implementation method thereof | |
| CN108509326B (en) | Service state statistical method and system based on nginx log | |
| CN105956082A (en) | Real-time data processing and storage system | |
| CN110519077A (en) | A kind of Log Collect System, method, apparatus, server and storage medium | |
| CN109542593B (en) | NIFI-based data processing flow design method | |
| CN103595577B (en) | System and method for monitoring whether traffic of interconnection interfaces between ISPs exceeds threshold valve | |
| CN105930502B (en) | System, client and method for collecting data | |
| CN107729219A (en) | Resource monitoring method, device and terminal based on super fusion storage system | |
| US20160021188A1 (en) | Generic Network Trace with Distributed Parallel Processing and Smart Caching | |
| CN103853826A (en) | Distributed type performance data processing method | |
| CA3010530A1 (en) | Single well cross steam and gravity drainage (sw-xsagd) | |
| CN103856354A (en) | Method for achieving unified management of logs of cluster storage system | |
| WO2015187001A3 (en) | System and method for managing resources failure using fast cause and effect analysis in a cloud computing system | |
| CN106599120A (en) | Stream processing framework-based data processing method and apparatus | |
| WO2015139565A1 (en) | Heterogeneous logging system management configuration | |
| US10001940B2 (en) | Limiting memory in a distributed environment at an operator or operator grouping level | |
| CN105553691A (en) | Monitoring alarm realization method based on electric power supervision information | |
| US20170010948A1 (en) | Monitoring a computing environment | |
| CN106817262A (en) | A kind of log analysis device | |
| KR101600275B1 (en) | Real-Time Big Data Event Processing System | |
| CN103685602A (en) | In-band out-band multiple-address associating method bringing convenience to node management | |
| CN109828968B (en) | Data deduplication processing method, device, equipment, cluster and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20141022 |
|
| RJ01 | Rejection of invention patent application after publication |