CN112948056A - Log log collection method and system on edge node for kubenge - Google Patents
Log log collection method and system on edge node for kubenge Download PDFInfo
- Publication number
- CN112948056A CN112948056A CN202110243893.2A CN202110243893A CN112948056A CN 112948056 A CN112948056 A CN 112948056A CN 202110243893 A CN202110243893 A CN 202110243893A CN 112948056 A CN112948056 A CN 112948056A
- Authority
- CN
- China
- Prior art keywords
- log
- node
- module
- logs
- kubbeedge
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/176—Support for shared access to files; File sharing support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/069—Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/45595—Network integration; Enabling network access in virtual machine instances
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Software Systems (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a method and a system for collecting log logs on an edge node aiming at kubbeelde, belonging to the field of kubberenets; the method specifically comprises the following steps: s1, running log proxy at Edge node to save self-made Edge node log; s2 collecting and recording log, naming the container as log-agent; s3, the log-agent sends all logs collected by the kubbeelde self-system to an Elasticsearch at the back end; s4, collecting error logs by an edge node when the kubbedge node is self-made; the mode realized by the invention can continuously collect the log after the kubendge node is self-made, and can upload the collected log when the kubendge node is accessed to the kubennettes cluster again (elastic search), thereby ensuring the continuity and persistence of the kubendge log.
Description
Technical Field
The invention discloses a method and a system for collecting log logs on an edge node aiming at kubbeelde, and relates to the technical field of kubberenets.
Background
Kubeedge manufactured a technique to collect logs independently and upload them after connection to kubberenets clusters: the method is a software architecture technology, and realizes how to continuously collect logs after edge is self-made offline, and the stored logs are uploaded to E (elastic search) after being accessed into a kubernets cluster again, and are displayed through K (Kibana).
The Elasticissearch is a distributed, high-expansion and high-real-time search and data analysis engine. It can conveniently make a large amount of data have the capability of searching, analyzing and exploring. The horizontal flexibility of the elastic search is fully utilized, so that the data becomes more valuable in a production environment.
Kibana is a Web interface for log analysis provided by Logstash and ElasticSearch. The method can be used for carrying out various operations such as efficient searching, visualization, analysis and the like on the log.
When the traditional kubbeelde node is connected with a kubberenets cluster, data collection can be carried out through docker json and fluent, and unified data collection processing is carried out so as to support log persistence, alarm and the like. When the kubbeedge is connected with the kubberenets cluster, the logs are collected by a fluent agent of a certain node of the host container log directory mapping/var/lib/contacts fluent-elastic search through receiving the logs generated by the docker logging driver. After the kubenedge node is self-made, the fluent-elastic search cannot be mapped to/var/log/contianers of the self-made node, and the log of the self-made node cannot be collected. Therefore, the new challenge is brought, if the log of the edge node is collected by rolling continuously when the kubbeelde node is homemade, and the log received when the kubbeelde node is connected to the kubberenets cluster again is uploaded to the homemade cluster, and the log is displayed by K (Kibana) to E (elastic search).
Disclosure of Invention
Aiming at the problems in the prior art, the invention provides a method and a system for collecting log logs on an edge node of a kubendge, and the adopted technical scheme is as follows: a method for collecting log logs on an edge node of a kubenge specifically comprises the following steps:
s1, running log proxy at Edge node to save self-made Edge node log;
s2 collecting and recording log, naming the container as log-agent;
s3, the log-agent sends all logs collected by the kubbeelde self-system to an Elasticsearch at the back end;
and when the S4 kubbedge node is self-made, the edge node collects the error log.
The method for collecting the logs by the kubbeedge node in the S3 specifically comprises the following steps:
s301, running a log collection service on the Kubedge node;
s302, storing and collecting logs on the Kubedge node when the Kubedge node is self-made;
s303 uploads logs to the shared log volume when kubbedge reconnects to the k8S cluster.
The method for collecting the logs by the kubbeedge node in the S3 specifically comprises the following steps:
s311 kubbedge node runs log collection service and provides shared log volume
S312, when the kubbeedge node is autonomous, the logs on the node are switched and stored to the local;
s313 uploads logs to the shared log volume when kubbeedge reconnects to the k8S cluster.
The method for collecting the logs by the kubbeedge node in the S3 specifically comprises the following steps:
s321, transmitting and displaying the log on each kubbeedge node through a customized network;
s322, running a pod real-time analysis display log at the kubbeedge node;
s323, storing and collecting logs on the kubbeedge node when the kubbeedge node is autonomous;
s324, when the kubbeedge is reconnected to the k8S cluster, analyzing the presentation log according to the collection time.
A log collection system on an edge node aiming at kubene specifically comprises a log storage module, a log naming module, a log delivery module and a log feedback module:
a log saving module: running log proxy at Edge node to store self-made log of Edge node;
a log naming module: the container class for collecting and recording the log logs is named log-agent;
a log submitting module: the log-agent provides all logs collected by the kubbeelde self-system to an Elasticsearch at the back end;
a log feedback module: and when the kubbedge node is self-made, the edge node collects the error log.
The log submitting module specifically comprises an operation module A, a collection module A and an uploading module A:
an operation module: running a log collection service on the Kubeedge node;
a collection module A: saving and collecting logs on the Kubeedge node when the Kubeedge node is self-made;
and an uploading module A: the journal is uploaded to the shared journal volume when kubbeedge reconnects to the k8s cluster.
The log submitting module specifically comprises a collecting module B, a switching module B and an uploading module B:
and a collection module B: running log collection service at kubbeedge node and providing shared log volume
A switching module B: switching logs on the kubbeedge nodes during the self-governing of the kubbeedge nodes and storing the logs to the local;
and an uploading module B: the journal is uploaded to the shared journal volume when kubbeedge reconnects to the k8s cluster.
The log submitting module specifically comprises a transmission module, an operation module C, a collection module C and a display module:
a transmission module: the log on each kubbeedge node is transmitted and displayed through a customized network;
and an operation module C: running a pod real-time analysis display log at the kubbeedge node;
a collection module C: saving and collecting logs on the kubbeedge nodes when the kubbeedge nodes are autonomous;
a display module: the presentation logs were analyzed according to collection time when kubbedge reconnected to the k8s cluster.
The invention has the beneficial effects that: the mode realized by the invention can continuously collect the log after the kubendge node is self-made, and can upload the collected log when the kubendge node is accessed to the kubennettes cluster again (elastic search), thereby ensuring the continuity and persistence of the kubendge log.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
FIG. 1 is a flow chart of the method of the present invention; fig. 2 is a schematic diagram of the system of the present invention.
Detailed Description
The present invention is further described below in conjunction with the following figures and specific examples so that those skilled in the art may better understand the present invention and practice it, but the examples are not intended to limit the present invention.
The first embodiment is as follows: a
A method for collecting log logs on an edge node of a kubenge specifically comprises the following steps:
s1, running log proxy at Edge node to save self-made Edge node log;
s2 collecting and recording log, naming the container as log-agent;
s3, the log-agent sends all logs collected by the kubbeelde self-system to an Elasticsearch at the back end;
s4, collecting error logs by an edge node when the kubbedge node is self-made;
when the method is used, firstly, the Kubeedge node runs a node-level log proxy at the Edge node after self-making to store logs of the Edge node after self-making, a container class for collecting and recording the log is named log-agent, a secondary container can store log logs of all containers on all the Edge nodes after self-making of the Edge node, so that the log-agent must run on each Edge node, and when the Kubeedge node is added into the Kubeedge cluster again, the log-agent can provide collected logs to an elastic search at the back end, and the log-agent can be customized fluent; the contents of collected logs are different between the Kubeedge node online mode and the Kubeedge node self-control mode, when the Kubeedge node online mode is the collected logs are complete, the edge node only collects error logs when the Kubeedge node self-control mode is carried out, and therefore the self-control log collection cost is reduced;
further, the method for collecting the log by the kubbeedge node in the S3 specifically includes:
s301, running a log collection service on the Kubedge node;
s302, storing and collecting logs on the Kubedge node when the Kubedge node is self-made;
s303, uploading the log to a shared log volume when the kubbeedge is reconnected to the k8S cluster;
running a log collection service on the Kubeedge node, uploading logs on the Kubeedge node in real time or at regular time, storing and collecting the logs in the Kubeedge node when the Kubeedge node self-makes, storing and collecting the logs in the Kubeedge node, and uploading the logs stored during self-making when the Kubeedge node is connected to the k8s cluster again;
further, the method for collecting the log by the kubbeedge node in the S3 specifically includes:
s311 kubbedge node runs log collection service and provides shared log volume
S312, when the kubbeedge node is autonomous, the logs on the node are switched and stored to the local;
s313, when the kubbeedge is reconnected to the k8S cluster, uploading the log to the shared log volume;
the method comprises the steps that log collection service is operated and a shared log volume is provided, a kubendge node collects logs in real time and stores the logs into the shared log volume on line, when the kubendge node is self-made, the kubendge node switches the logs to be stored locally, and when the kubendge node is connected to a k8s cluster again, the logs stored during self-making are uploaded to the shared log volume;
still further, the method for collecting logs by the kubbeedge node in S3 specifically includes:
s321, transmitting and displaying the log on each kubbeedge node through a customized network;
s322, running a pod real-time analysis display log at the kubbeedge node;
s323, storing and collecting logs on the kubbeedge node when the kubbeedge node is autonomous;
s324, when the kubbeedge is reconnected to the k8S cluster, analyzing the display log according to the collection time;
the logs on the Kubeedge nodes are collected and displayed through customized network distribution, the logs of each node are transmitted and displayed through a customized network, the Kubeedge nodes operate one pod, the Kubeedge nodes analyze and display the logs in real time online, when the Kubeedge nodes are self-made, the logs are stored and collected locally on the Kubeedge nodes, the logs are stored and collected on the Kubeedge nodes, and when the Kubeedge nodes are connected to the k8s cluster again, the logs can be analyzed and displayed according to the collection time of the logs.
Example two:
a log collection system on an edge node aiming at kubene specifically comprises a log storage module, a log naming module, a log delivery module and a log feedback module:
a log saving module: running log proxy at Edge node to store self-made log of Edge node;
a log naming module: the container class for collecting and recording the log logs is named log-agent;
a log submitting module: the log-agent provides all logs collected by the kubbeelde self-system to an Elasticsearch at the back end;
a log feedback module: and when the kubbedge node is self-made, the edge node collects the error log.
Further, the log submitting module specifically comprises an operation module a, a collection module a and an uploading module a:
an operation module: running a log collection service on the Kubeedge node;
a collection module A: saving and collecting logs on the Kubeedge node when the Kubeedge node is self-made;
and an uploading module A: the journal is uploaded to the shared journal volume when kubbeedge reconnects to the k8s cluster.
Further, the log submitting module specifically includes a collecting module B, a switching module B, and an uploading module B:
and a collection module B: running log collection service at kubbeedge node and providing shared log volume
A switching module B: switching logs on the kubbeedge nodes during the self-governing of the kubbeedge nodes and storing the logs to the local;
and an uploading module B: the journal is uploaded to the shared journal volume when kubbeedge reconnects to the k8s cluster.
Further, the log submitting module specifically comprises a transmission module, an operation module C, a collection module C and a display module:
a transmission module: the log on each kubbeedge node is transmitted and displayed through a customized network;
and an operation module C: running a pod real-time analysis display log at the kubbeedge node;
a collection module C: saving and collecting logs on the kubbeedge nodes when the kubbeedge nodes are autonomous;
a display module: analyzing the presentation log according to the collection time when the kubbeedge reconnects to the k8s cluster;
the mode realized by the invention can continuously collect the log after the kubendge node is self-made, and can upload the collected log when the kubendge node is accessed to the kubennettes cluster again (elastic search), thereby ensuring the continuity and persistence of the kubendge log.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (8)
1. A method for collecting log logs on an edge node of a kubene is characterized by specifically comprising the following steps:
s1, running log proxy at Edge node to save self-made Edge node log;
s2 collecting and recording log, naming the container as log-agent;
s3, the log-agent sends all logs collected by the kubbeelde self-system to an Elasticsearch at the back end;
and when the S4 kubbedge node is self-made, the edge node collects the error log.
2. The method according to claim 1, wherein the method for collecting logs by the kubbeelde node in S3 specifically includes:
s301, running a log collection service on the Kubedge node;
s302, storing and collecting logs on the Kubedge node when the Kubedge node is self-made;
s303 uploads logs to the shared log volume when kubbedge reconnects to the k8S cluster.
3. The method according to claim 1, wherein the method for collecting logs by the kubbeelde node in S3 specifically includes:
s311 kubbedge node runs log collection service and provides shared log volume
S312, when the kubbeedge node is autonomous, the logs on the node are switched and stored to the local;
s313 uploads logs to the shared log volume when kubbeedge reconnects to the k8S cluster.
4. The method according to claim 1, wherein the method for collecting logs by the kubbeelde node in S3 specifically includes:
s321, transmitting and displaying the log on each kubbeedge node through a customized network;
s322, running a pod real-time analysis display log at the kubbeedge node;
s323, storing and collecting logs on the kubbeedge node when the kubbeedge node is autonomous;
s324, when the kubbeedge is reconnected to the k8S cluster, analyzing the presentation log according to the collection time.
5. A log collection system on an edge node aiming at kubene is characterized by specifically comprising a log storage module, a log naming module, a log submitting module and a log feedback module:
a log saving module: running log proxy at Edge node to store self-made log of Edge node;
a log naming module: the container class for collecting and recording the log logs is named log-agent;
a log submitting module: the log-agent provides all logs collected by the kubbeelde self-system to an Elasticsearch at the back end;
a log feedback module: and when the kubbedge node is self-made, the edge node collects the error log.
6. The system of claim 5, wherein the log submitting module specifically comprises an operation module a, a collection module a, and an uploading module a:
an operation module: running a log collection service on the Kubeedge node;
a collection module A: saving and collecting logs on the Kubeedge node when the Kubeedge node is self-made;
and an uploading module A: the journal is uploaded to the shared journal volume when kubbeedge reconnects to the k8s cluster.
7. The system according to claim 5, wherein the log submitting module specifically includes a collecting module B, a switching module B, and an uploading module B:
and a collection module B: running log collection service at kubbeedge node and providing shared log volume
A switching module B: switching logs on the kubbeedge nodes during the self-governing of the kubbeedge nodes and storing the logs to the local;
and an uploading module B: the journal is uploaded to the shared journal volume when kubbeedge reconnects to the k8s cluster.
8. The system as claimed in claim 5, wherein the log submitting module specifically comprises a transmission module, an operation module C, a collection module C and a presentation module:
a transmission module: the log on each kubbeedge node is transmitted and displayed through a customized network;
and an operation module C: running a pod real-time analysis display log at the kubbeedge node;
a collection module C: saving and collecting logs on the kubbeedge nodes when the kubbeedge nodes are autonomous;
a display module: the presentation logs were analyzed according to collection time when kubbedge reconnected to the k8s cluster.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110243893.2A CN112948056A (en) | 2021-03-05 | 2021-03-05 | Log log collection method and system on edge node for kubenge |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110243893.2A CN112948056A (en) | 2021-03-05 | 2021-03-05 | Log log collection method and system on edge node for kubenge |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112948056A true CN112948056A (en) | 2021-06-11 |
Family
ID=76247831
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110243893.2A Pending CN112948056A (en) | 2021-03-05 | 2021-03-05 | Log log collection method and system on edge node for kubenge |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112948056A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8069366B1 (en) * | 2009-04-29 | 2011-11-29 | Netapp, Inc. | Global write-log device for managing write logs of nodes of a cluster storage system |
| CN109245931A (en) * | 2018-09-19 | 2019-01-18 | 四川长虹电器股份有限公司 | The log management of container cloud platform based on kubernetes and the implementation method of monitoring alarm |
| CN110955578A (en) * | 2019-11-26 | 2020-04-03 | 深圳市信联征信有限公司 | Log collection method and device based on host machine, computer equipment and storage medium |
| CN111866707A (en) * | 2019-12-31 | 2020-10-30 | 北京骑胜科技有限公司 | Equipment offline processing method and device |
| CN112069145A (en) * | 2020-08-31 | 2020-12-11 | 北京易捷思达科技发展有限公司 | Log collection system and method suitable for large-scale cloud deployment environment |
-
2021
- 2021-03-05 CN CN202110243893.2A patent/CN112948056A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8069366B1 (en) * | 2009-04-29 | 2011-11-29 | Netapp, Inc. | Global write-log device for managing write logs of nodes of a cluster storage system |
| CN109245931A (en) * | 2018-09-19 | 2019-01-18 | 四川长虹电器股份有限公司 | The log management of container cloud platform based on kubernetes and the implementation method of monitoring alarm |
| CN110955578A (en) * | 2019-11-26 | 2020-04-03 | 深圳市信联征信有限公司 | Log collection method and device based on host machine, computer equipment and storage medium |
| CN111866707A (en) * | 2019-12-31 | 2020-10-30 | 北京骑胜科技有限公司 | Equipment offline processing method and device |
| CN112069145A (en) * | 2020-08-31 | 2020-12-11 | 北京易捷思达科技发展有限公司 | Log collection system and method suitable for large-scale cloud deployment environment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111077870A (en) | Intelligent OPC data real-time acquisition and monitoring system and method based on stream calculation | |
| CN112187514A (en) | Intelligent operation and maintenance system, method and terminal for data center network equipment | |
| CN101873005B (en) | Method for realizing balanced acquisition of electric energy | |
| CN109151072A (en) | A kind of edge calculations system based on mist node | |
| CN101902497B (en) | Cloud computing based internet information monitoring system and method | |
| CN103761309A (en) | Operation data processing method and system | |
| CN103326896B (en) | The system and method for the information data that a kind of user of collection produces on the internet | |
| US20230306042A1 (en) | Management method and apparatus for internet of things device, medium, and electronic device | |
| CN113176948B (en) | Edge gateway, edge computing system and configuration method thereof | |
| CN110995859A (en) | Intelligent transformer substation supporting platform system based on ubiquitous Internet of things | |
| CN112101589B (en) | Ship remote technical support system based on cloud computing | |
| CN105843936A (en) | Service data report form method and system | |
| CN109747684B (en) | Comprehensive monitoring system and method for rail transit and computer equipment | |
| JP2023506239A (en) | Systems and methods for autonomous monitoring and recovery in hybrid energy management | |
| CN115102986B (en) | Internet of things data distribution and storage method and system in edge environment | |
| CN108009089B (en) | Incremental machine learning method and system based on transparent computing | |
| CN112948056A (en) | Log log collection method and system on edge node for kubenge | |
| CN208890843U (en) | A kind of edge calculations system based on mist node | |
| CN113127526A (en) | Distributed data storage and retrieval system based on Kubernetes | |
| CN201726426U (en) | Internet information monitoring system based on cloud computing | |
| CN115357459A (en) | Log collecting method based on kubernets | |
| KR20220071548A (en) | Sensor data processing system | |
| CN102854867A (en) | Centralized control method for operation of vehicle-mounted devices of satellite communication vehicle | |
| CN112765294A (en) | Meteorological big data processing and scheduling system | |
| CN215216639U (en) | Central air conditioning intelligence system of patrolling and examining for cigarette factory |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210611 |