CN109660397A - For acquiring system, the method and apparatus of log - Google Patents
For acquiring system, the method and apparatus of log Download PDFInfo
- Publication number
- CN109660397A CN109660397A CN201811574616.4A CN201811574616A CN109660397A CN 109660397 A CN109660397 A CN 109660397A CN 201811574616 A CN201811574616 A CN 201811574616A CN 109660397 A CN109660397 A CN 109660397A
- Authority
- CN
- China
- Prior art keywords
- log
- server
- cluster
- application
- acquired
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/069—Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
Abstract
The embodiment of the present application discloses system, method and apparatus for acquiring log.One specific embodiment of the system includes: including at least one kubernetes cluster, caas cluster, log collection configuration server, log storage server and O&M server, kubernetes cluster includes host node and at least one working node, and kafka cluster application is equipped in host node.The system for being used to acquire log realizes the diversified log collection in source, the scalable appearance bring configuration information of dynamic more new container elasticity, and automatically updates the configuration information of kubernetes cluster dimension variation.
Description
Technical field
The invention relates to field of computer technology, and in particular to for acquiring system, the method and apparatus of log.
Background technique
In cloud platform, Log Administration System is important cloud service infrastructure, can be used for monitor supervision platform operation shape
Condition, Check System failure.And the missing and inaccuracy of daily record data can make troubles for work such as log analysis, orientation problems.
Traditional log collection can dispose acquisition client on every service server, acquire the log recording of client
Journal file in journal file, is then uploaded the log recording of certain format by module by the sending module of acquisition client
To log server.
In addition, the application with container technique in cloud computing platform, can also in advance by business service in a reservoir
Journal file path is stored in default container Log Directory.In preset container Log Directory, pass through service identifiers, container
Mark, journal file mark acquire corresponding log recording.
Summary of the invention
The embodiment of the present application proposes system, method and apparatus for acquiring log.
In a first aspect, the embodiment of the present application provides a kind of system for acquiring log, including at least one
Kubernetes cluster, caas cluster, log collection configuration server, log storage server and O&M server,
Kubernetes cluster includes host node and at least one working node, and kafka cluster application is equipped in host node, in which:
Host node is configured to: acquiring the component log of each kubernetes component in the host node, and by component collected
Log is sent to kafka cluster indicated by the kafka cluster application in the host node；It is corresponding in response to listening to the host node
Working node in container change, according to current each appearance in the changed working node of the corresponding container of the host node
Device state generates the log collection configuration information of relevant work node and is sent to log collection configuration server；Working node
It is configured to: obtaining the log collection configuration of the working node from log collection configuration server every the first prefixed time interval
Information；According to acquired log collection configuration information, the log of kubernetes component and the container day of the working node are acquired
Will, and log collected is sent to indicated by the kafka cluster application installed in the corresponding host node of the working node
Kafka cluster；Caas cluster is configured to: the component log of each Caas component in acquisition Caas cluster, and will be collected
Component log is sent to O&M server；Log storage server is configured to: every the second prefixed time interval from Caas collection
The corresponding log of kafka cluster application installed in the host node for each kubernetes cluster that group's acquisition Caas cluster is managed
Read configuration information；Log is pulled according to position indicated by acquired log read configuration information and is stored；O&M service
Device is configured to: obtaining the log stored in log storage server；Obtain the log stored in O&M server；And it is in
Existing acquired log.
In some embodiments, ElasticSearch application is installed in O&M server；And O&M server also by
It is configured to: being applied using the ElasticSearch installed and clustering index is carried out to acquired log.
In some embodiments, Kibana application is installed in O&M server；And O&M server is configured to: being in
Existing acquired log, comprising: O&M server is configured to: utilize installed Kibana application that acquired day is presented
Will.
Second aspect, the embodiment of the present application provide a kind of method for acquiring log, applied to for acquiring log
System in O&M server, wherein the system for being used to acquire log includes at least one kubernetes cluster, caas
Cluster, log collection configuration server, log storage server and O&M server, kubernetes cluster include host node and
At least one working node is equipped with kafka cluster application in host node, this method comprises: obtaining the log storage service
The log stored in device；Obtain the log stored in O&M server；And acquired log is presented.
In some embodiments, ElasticSearch application is installed in O&M server；And method further include: benefit
It is applied with the ElasticSearch installed and clustering index is carried out to acquired log.
In some embodiments, Kibana application is installed in O&M server；And acquired log is presented, packet
It includes: utilizing installed Kibana application that acquired log is presented.
The third aspect, the embodiment of the present application provide it is a kind of for acquiring the device of log, applied to for acquiring log
System in O&M server, wherein the system for being used to acquire log includes at least one kubernetes cluster, caas
Cluster, log collection configuration server, log storage server and O&M server, kubernetes cluster include host node and
At least one working node is equipped with kafka cluster application in host node, which includes: first acquisition unit, is configured to
Obtain the log stored in log storage server；Second acquisition unit, is configured to obtain and is stored in O&M server
Log；Display unit is configured to present acquired log.
In some embodiments, wherein ElasticSearch application is installed in O&M server；And device also wraps
Include: indexing units are configured to be applied using the ElasticSearch installed to acquired log progress clustering index.
In some embodiments, Kibana application is installed in O&M server；And display unit is further configured
At: utilize installed Kibana application that acquired log is presented.
Fourth aspect, the embodiment of the present application provide a kind of O&M server, comprising: one or more processors；Storage
Device is stored thereon with one or more programs, when said one or multiple programs are executed by said one or multiple processors
When, so that said one or multiple processors realize the method as described in implementation any in second aspect.
5th aspect, the embodiment of the present application provide a kind of computer readable storage medium, are stored thereon with computer journey
Sequence, wherein realized when the computer program is executed by one or more processors such as implementation description any in second aspect
Method.
Traditional log collection method and existing container log collection method are unable to satisfy the day of extensive container service
Will acquires demand.These methods have the following deficiencies:
1, existing log collection tool needs to carry out manual configuration, acquisition mode and log path including log.This
It requires to complete after application deployment.And there are dynamic drift and elasticity are scalable for the container instance in container cloud platform
The characteristic of appearance, traditional sampling instrument can not perceive the variation of this kind of configuration information.
2, existing log collection tool can not log source in clear positioning vessel cloud.Business in container cloud platform
Usually by multiple pod, (Pod is minimum/simplest basic unit of Kubernetes creation or deployment, a Pod generation for service
A process being currently running on table cluster) carrying, this needs sampling instrument to be able to record the mark in log source, including calculates
The corresponding pod of node, service application, Service Instance and container.And need to comprehensively consider business service log and platform certainly
The acquisition of body log.
It is provided by the embodiments of the present application for acquiring system, the method and apparatus of log, by by caas cluster management extremely
A few kubernetes cluster, kubernetes cluster includes host node and at least one working node, is installed in host node
There is kafka cluster application, the host node in kubernetes cluster acquires the group of each kubernetes component in the host node
Part log, and component log collected is sent to kafka collection indicated by the kafka cluster application in the host node
Group；And there are container variations in the corresponding each working node of the host node in response to listening to for host node, according to the host node
Current each container state in corresponding each working node generates the log collection configuration information of relevant work node and is sent to
Log collection configuration server；Working node in kubernetes, which is configured every the first prefixed time interval from log collection, to be taken
Business device obtains the log collection configuration information of the working node；And according to acquired log collection configuration information, acquisition should
The log of kubernetes component and container log of working node, and it is corresponding that log collected is sent to the working node
Kafka cluster indicated by the kafka cluster application installed in host node；Caas cluster acquires each Caas component in Caas cluster
Component log, and component log collected is sent to O&M server；Log storage server is default every second
The kafka collection that time interval is installed from the host node that Caas cluster obtains each kubernetes cluster that Caas cluster is managed
Group applies corresponding log read configuration information；It is drawn from kafka cluster indicated by acquired log read configuration information
It takes log and stores；O&M server obtains the log stored in log storage server；It obtains and is stored in O&M server
Log；And acquired log is presented.It is as follows present in existing Log Collect System to solve the problems, such as: the first,
Log source Diversification Problem, this system can acquire the log of Caas cluster, each kubernetes in kubernetes cluster
Component and container log；The second, the scalable appearance bring configuration information dynamic replacement problem of container elasticity, this system do not need manually
The configuration information of acquisition log is set；The configuration replacement problem of third, kubernetes cluster dimension variation, this system do not need
The configuration information of manual setting reading log.
Detailed description of the invention
By reading a detailed description of non-restrictive embodiments in the light of the attached drawings below, the application's is other
Feature, objects and advantages will become more apparent upon:
Fig. 1 is that one embodiment of the application can be applied to exemplary system architecture figure therein；
Fig. 2 is the timing diagram according to one embodiment of the system for acquiring log of the application；
Fig. 3 is the flow chart according to one embodiment of the method for acquiring log of the application；
Fig. 4 is the structural schematic diagram according to one embodiment of the device for acquiring log of the application；
Fig. 5 is adapted for the structural schematic diagram for the computer system for realizing the O&M server of the embodiment of the present application.
Specific embodiment
The application is described in further detail with reference to the accompanying drawings and examples.It is understood that this place is retouched
The specific embodiment stated is used only for explaining related invention, rather than the restriction to the invention.It also should be noted that in order to
Convenient for description, part relevant to related invention is illustrated only in attached drawing.
It should be noted that in the absence of conflict, the features in the embodiments and the embodiments of the present application can phase
Mutually combination.The application is described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.
Fig. 1 is shown can be using the exemplary system architecture of the embodiment of the system for acquiring log of the application
100。
As shown in Figure 1, system architecture 100 may include caas cluster 101, kubernetes (being also " k8s ") cluster
102,103,104, log collection configuration server 105, log storage server 106, O&M server 107 and network 108,
K8s cluster 102 include host node 1021 and working node 1022,1023,1024, k8s cluster 102 include host node 1031 and work
Make node 1032,1033,1034, k8s cluster 104 include host node 1041 and working node 1042,1043,1044, host node
1021, kafka cluster application is mounted in 1031 and 1041, respectively correspond corresponding kafka cluster.
Network 108 is in caas cluster 101, k8s cluster 102,103,104, log collection configuration server 105, day
The medium of communication link is provided between will storage server 106 and O&M server 107.Network 108 may include various connections
Type, such as wired, wireless communication link or fiber optic cables etc..
It should be noted that in some cases, log storage server 106 and k8s cluster 102,103,104 can be located
In different network areas, i.e., Network Isolation between the two.
User can be used caas cluster 101 and be interacted by network 108 with k8s cluster 102,103,104, to receive or send out
Send message etc..Caas cluster platform application can be installed, for creating and managing k8s cluster on Caas cluster 101.
O&M server 107 can be hardware, be also possible to software.When O&M server 107 is hardware, tool can be
There are the various electronic equipments of display screen, including but not limited to smart phone, tablet computer, pocket computer on knee and desk-top meter
Calculation machine etc..When O&M server 107 is software, may be mounted in above-mentioned cited electronic equipment.It may be implemented
At multiple softwares or software module (such as service is presented for providing log), single software or software mould also may be implemented into
Block.It is not specifically limited herein.
It should be noted that for acquiring the method for log generally by O&M server provided by the embodiment of the present application
107 execute, and correspondingly, the device for acquiring log is generally positioned in O&M server 107.
It should be noted that log collection configuration server 105 and log storage server 106 can be hardware, it can also
To be software.When log acquisition configuration server 105 and log storage server 106 are hardware, multiple clothes may be implemented into
The distributed server cluster of business device composition, also may be implemented into individual server.When log acquisition configuration server 105 and day
When will storage server 106 is software, multiple softwares or software module may be implemented into, single software or soft also may be implemented into
Part module.It is not specifically limited herein.
Host node 1021,1031,1041 and working node 1022,1023,1024,1032,1033,1034,1042,
1043, it 1044 can be hardware (for example, physical machine), be also possible to software (for example, virtual machine).When host node 1021,1031,
1041 and working node 1022,1023,1024,1032,1033,1034,1042,1043,1044 be hardware when, may be implemented into
The distributed server collection of multiple server compositions, also may be implemented into individual server.When above-mentioned host node 1021,1031,
1041 and working node 1022,1023,1024,1032,1033,1034,1042,1043,1044 be software when, may be implemented into
Multiple softwares or software module (such as providing Distributed Services), also may be implemented into single software or software module.?
This is not specifically limited.
It should be noted that Caas cluster 101 can be hardware, it is also possible to software.When Caas cluster 101 is hardware
When, the distributed server collection of multiple server compositions may be implemented into, individual server also may be implemented into.As above-mentioned Caas
When cluster 101 is software, multiple softwares or software module (such as providing Distributed Services) may be implemented into, it can also be real
Ready-made single software or software module.It is not specifically limited herein.
It should be understood that the number of terminal device, network and server in Fig. 1 is only schematical.According to realization need
It wants, can have any number of terminal device, network and server.
With continued reference to Fig. 2, it illustrates the timing according to one embodiment of the system for acquiring log of the application
200。
The system for acquiring log in the embodiment of the present application may include at least one kubernetes cluster, caas
Cluster, log collection configuration server, log storage server and O&M server, kubernetes cluster include host node and
At least one working node is equipped with kafka cluster application in host node.
As shown in Fig. 2, may include according to the timing 200 of one embodiment of the system for acquiring log of the application
Following steps:
Step 201, host node acquires the component log of each kubernetes component in the host node, and will be acquired
Component log be sent to kafka cluster indicated by the kafka cluster application in the host node.
In the present embodiment, the host node (that is, master) in kubernetes cluster can acquire the host node in real time
In each kubernetes component component log, and kafka component log collected being sent in the host node
(Kafka is the open source stream process platform developed by Apache Software Foundation to cluster application, by Scala and written in Java.
Kafka is that a kind of distributed post of high-throughput subscribes to message system, it can handle the institute in the website of consumer's scale
Have movement flow data) indicated by kafka cluster.
In the present embodiment, kafka cluster application can be installed in the host node in kubernetes cluster, due to
Kafka cluster application is used to manage kafka cluster, and therefore, the kafka cluster installed in the host node of kubernetes cluster is answered
With can indicate kafka cluster, kafka indicated by the kafka cluster application installed in the host node of kubernetes cluster
Cluster may include the corresponding physical machine of host node of the kubernetes cluster, can not also include the kubernetes cluster
The corresponding physical machine of host node.
In practice, to the host node in each kubernetes cluster, if the operating system of the host node is
Centos6.3, then the host node can directly from the host node /var/log/kubernetes/ catalogue under acquire the master
The component log of each kubernetes component in node；, whereas if the operating system of the host node is not centos6.3,
Then the host node can acquire the component log of each kubernetes component in the host node from journald.
In practice, since ks8s cluster is often with log storage server Network Isolation, therefore, it is necessary to by host node
Log collected is sent to kafka collection indicated by the kafka cluster application installed in the corresponding host node of the working node
Group.
Step 202, host node changes in response to listening to container in the corresponding working node of the host node, according to this
Current each container state in the changed working node of the corresponding container of host node, the log for generating relevant work node are adopted
Collection configuration information is simultaneously sent to log collection configuration server.
In the present embodiment, if container changes in working node in kubernetes cluster, for example, container increases
Add or reduce, then the corresponding host node of the working node can monitor this variation, and in the appearance for listening to the working node
After device variation, according to current each container state of the changed working node of the container, the day of relevant work node is generated
Will acquisition configuration information is simultaneously sent to log collection configuration server.
In practice, host node can be according to current each container state of the changed working node of the container, according to pre-
If log path format, generate the log collection configuration information of relevant work node and be sent to log collection configuration service
Device.It wherein, include the identification information of container in preset log path format, if container changes, container identification
It will change, therefore, it is necessary in the case where container changes, by new log collection configuration information (for example, log
Acquisition routing information) it is sent to log collection configuration server.
Step 203, working node obtains the work section from log collection configuration server every the first prefixed time interval
The log collection configuration information of point.
Since container is dynamic change in working node, in order to guarantee that working node can be accurately obtained the work section
The log of kubernetes component and container log of point, working node can match every the first prefixed time interval from log collection
Set the log collection configuration information that server obtains the working node.
Here, the first prefixed time interval can be preassigned by technical staff.It is pre- that setting first can also be provided
If the interface of time interval, and receive time interval that user inputs in the interface as the first prefixed time interval.
Step 204, working node acquires the working node according to acquired log collection configuration information
The log of kubernetes component and container log, and log collected is sent in the corresponding host node of the working node and is pacified
Kafka cluster indicated by the kafka cluster application of dress.
In the present embodiment, working node can be somebody's turn to do according to log collection configuration information acquired in step 203, acquisition
The log of kubernetes component and container log of working node, and it is corresponding that log collected is sent to the working node
Kafka cluster indicated by the kafka cluster application installed in host node.
In practice, since ks8s cluster is often with log storage server Network Isolation, therefore, it is necessary to the sections that will work
Point log collected is sent to kafka indicated by the kafka cluster application installed in the corresponding host node of the working node
Cluster.
Step 205, in Caas cluster acquisition Caas cluster each Caas component component log, and by component collected
Log is sent to O&M server.
In the present embodiment, Caas cluster can directly acquire the component log of each Caas component in the Caas cluster, with
And component log collected is sent to O&M server.
In practice, Caas cluster can be adopted directly from the journal file under the fixation catalogue of the Caas electronic equipment run
Collect the component log of each Caas component in the Caas cluster.
Step 206, log storage server obtains Caas cluster from Caas cluster every the second prefixed time interval and is managed
The corresponding log read configuration information of kafka cluster application installed in the host node of each kubernetes cluster of reason.
In the present embodiment, since Caas cluster is responsible for creating and manage each kubernetes cluster, in Caas cluster
The cluster information for saving the Caas cluster can specifically include the cluster of kubernetes cluster included by the Caas cluster
Information, and the cluster information of kubernetes cluster may include the host node of host node included by the kubernetes cluster
Information, the host node information of host node may include that the corresponding log read of kafka cluster application installed on the host node is matched
Confidence breath, that is, the kafka installed on the host node applies in indicated kafka cluster what log where be stored with.
In this way, in order to accurately read the log acquired in kafka cluster, log storage server can be default every second
The kafka collection that time interval is installed from the host node that Caas cluster obtains each kubernetes cluster that Caas cluster is managed
Group applies corresponding log read configuration information.
Here, the second prefixed time interval can be preassigned by technical staff.It is pre- that setting second can also be provided
If the interface of time interval, and receive time interval that user inputs in the interface as the second prefixed time interval.
Step 207, log storage server position indicated by the acquired log read configuration information pulls log
And it stores.
In the present embodiment, log storage takes the log read configuration information that 207 business devices can be acquired from step 206
Indicated position pulls log and stores.
In practice, log read configuration information may include kafka server identification and corresponding day in kafka cluster
Will read path, in this way, log storage server can be from the log in server indicated by above-mentioned kafka server identification
It is drawn in read path and goes log and store.
By the record of above-mentioned steps 201 to step 204 and step 206 it is found that log storage server is drawn in step 207
It takes and the log stored may include: each in each host node for each kubernetes cluster that Caas cluster is managed
The component log of kubernetes component and the kubernetes component log of each working node and container log.
Step 208, O&M server obtains the log stored in log storage server.
In the present embodiment, O&M server mode can occur and obtain to store in log storage server using various
Log.Here, the log stored in log storage server may include for example, O&M server can be when third be preset
It is long to obtain the log stored in log storage server.
In another example the day that O&M server can also store in the acquisition log storage server for receiving user's sending
After the operational order of will, the log stored in log storage server is obtained.
As above-mentioned record it is found that the log acquired in the log storage server of O&M server may include Caas cluster
The component log of each kubernetes component and each work in each host node of each kubernetes cluster managed
The log of kubernetes component and container log of node.
Step 209, O&M server obtains the log stored in O&M server.
In the present embodiment, O&M server mode can be occurred and obtained the day stored in O&M server using various
Will.
For example, O&M server can obtain the log stored in O&M server every the 4th preset duration.
In another example the log that O&M server can also store in the acquisition O&M server for receiving user's sending
After operational order, the log stored in O&M server is obtained.
As above-mentioned record it is found that O&M server from log acquired in O&M server may include in Caas cluster it is each
The component log of Caas component.
Step 210, acquired log is presented in O&M server.
In the present embodiment, the day that O&M server can be acquired in rendering step 208 after executing the step 208
Will, log that can also be acquired in rendering step 209 after executing the step 209 can also first carry out step 208 again
After executing step 209, acquired log, can also first carry out step 209 certainly in rendering step 208 and step 209
After executing step 208 again, acquired log in rendering step 209 and step 208.
By step 210, O&M engineer can take the various logs seen in device and presented in O&M, thus realization pair
The monitoring for each k83 cluster that Caas cluster and Caas cluster are managed.
In some optional implementations of the present embodiment, ElasticSearch can be installed in freight charges server
Using.In this way before step 210, above-mentioned timing 200 can also include the following steps:
The ElasticSearch that O&M server by utilizing is installed, which is applied, carries out clustering index to acquired log.
In this way, when log being presented in step 210, can be convenient O&M engineer to various days after clustering index
Will carries out searching classification and checks.
In some optional implementations of the present embodiment, Kibana application is also equipped in O&M server,
In this way, step 210 can carry out as follows:
Acquired log is presented in the Kibana application that O&M server by utilizing is installed.
In this way, the visualization interface applied using Kibana, O&M engineer can more easily present acquired
Log.
The system provided by the above embodiment of the application includes at least one kubernetes cluster, caas cluster, log
Acquisition configuration server, the Log Collect System of log storage server and O&M server are mentioned by comprehensive log collection
O&M can be promoted optionally in combination with matched log searching, analysis tool and visualization interface for accurate log information
The working efficiency of engineer's Check System failure, supervisory control system running situation.
With continued reference to Fig. 3, it illustrates the processes according to one embodiment of the method for acquiring log of the application
300, applied to the O&M server in the system of acquisition log, wherein the system for acquiring log may include at least one
A kubernetes cluster, caas cluster, log collection configuration server, log storage server and O&M server,
Kubernetes cluster includes host node and at least one working node, and kafka cluster application is equipped in host node, this is used for
Acquire the process 300 of the method for log, comprising the following steps:
Step 301, the log stored in log storage server is obtained.
It in the present embodiment, can for acquiring the executing subject (such as O&M server shown in FIG. 1) of the method for log
Remotely to obtain the log stored in log storage server from log storage server.
Here, the concrete operations of step 301 and the operation of step 208 in embodiment shown in Fig. 2 are essentially identical, herein not
It repeats again.
Step 302, the log stored in O&M server is obtained.
In the present embodiment, the log stored in the available O&M server local of above-mentioned executing subject.
Here, the concrete operations of step 302 and the operation of step 209 in embodiment shown in Fig. 2 are essentially identical, herein not
It repeats again.
Step 303, acquired log is presented.
In the present embodiment, above-mentioned executing subject can use various implementation rendering steps
In the present embodiment, above-mentioned executing subject can be acquired in rendering step 301 after executing the step 301
Log, log that can also be acquired in rendering step 302 after executing the step 302 can also first carry out step 301
After executing step 302 again, acquired log, can also first carry out step certainly in rendering step 301 and step 302
After 302 execute step 302 again, acquired log in rendering step 302 and step 301.
In some optional implementations of the present embodiment, it can be equipped in above-mentioned executing subject
ElasticSearch application.In this way, above-mentioned process 300 can also include the following steps: before above-mentioned steps 303
It is applied using the ElasticSearch installed and clustering index is carried out to acquired log.
In this way, when log being presented in step 303, can be convenient O&M engineer to various days after clustering index
Will carries out searching classification and checks.
In some optional implementations of the present embodiment, Kibana is also equipped in above-mentioned executing subject and is answered
With in this way, step 303 may be carried out as follows:
Utilize installed Kibana application that acquired log is presented.
In this way, the visualization interface applied using Kibana, O&M engineer can more easily present acquired
Log.
Method provided by the embodiments of the present application for acquiring log obtains log storage server by O&M server
The component log of each kubernetes component and working node in host node in the kubernetes cluster of middle storage
The log of kubernetes component and container log obtain each Caas component in the Caas cluster that O&M server local is stored
Component log, and acquired log is presented, log collection diversification is realized, and improve O&M engineer and check system
The working efficiency for failure, the supervisory control system running situation of uniting.
With further reference to Fig. 4, as the realization to method shown in above-mentioned each figure, this application provides one kind for acquiring day
One embodiment of the device of will, applied to the O&M server in the system for acquiring log, wherein above-mentioned for acquiring
The system of log includes at least one kubernetes cluster, caas cluster, log collection configuration server, log storage service
Device and O&M server, kubernetes cluster include host node and at least one working node, are equipped with kafka in host node
Cluster application.The Installation practice is corresponding with embodiment of the method shown in Fig. 3, which specifically can be applied to various electronics
In equipment.
As shown in figure 4, the device 400 for acquiring log of the present embodiment may include: first acquisition unit 401, the
Two acquiring units 402 and display unit 403.Wherein, first acquisition unit 401 are configured to obtain above-mentioned log storage service
The log stored in device；Second acquisition unit 402 is configured to obtain the log stored in above-mentioned O&M server；It presents
Unit 403 is configured to present acquired log.
In the present embodiment, for acquiring first acquisition unit 401,402 and of second acquisition unit of the device 400 of log
The specific processing of display unit 403 and its brought technical effect can be respectively with reference to steps 301, step in Fig. 3 corresponding embodiment
Rapid 302 and step 303 related description, details are not described herein.
In some optional implementations of the present embodiment, it can be equipped in above-mentioned O&M server
ElasticSearch application；And above-mentioned apparatus 400 can also include: indexing units 404, be configured to utilize and be installed
ElasticSearch, which is applied, carries out clustering index to acquired log.
In some optional implementations of the present embodiment, Kibana can be installed in above-mentioned O&M server and answered
With；And above-mentioned display unit 303 can be further configured to: utilize installed Kibana application that acquired day is presented
Will.
It should be noted that it is provided by the embodiments of the present application for acquire in the device of log the realization details of each unit and
Technical effect can be with reference to the explanation of other embodiments in the application, and details are not described herein.
Below with reference to Fig. 5, it illustrates the computer systems for the O&M server for being suitable for being used to realize the embodiment of the present application
500 structural schematic diagram.O&M server shown in Fig. 5 is only an example, should not function to the embodiment of the present application and
Use scope brings any restrictions.
As shown in figure 5, computer system 500 includes central processing unit (CPU, Central Processing Unit)
501, it can be according to the program being stored in read-only memory (ROM, Read Only Memory) 502 or from storage section
508 programs being loaded into random access storage device (RAM, Random Access Memory) 503 and execute various appropriate
Movement and processing.In RAM 503, also it is stored with system 500 and operates required various programs and data.CPU 501,ROM
502 and RAM 503 is connected with each other by bus 504.Input/output (I/O, Input/Output) interface 505 is also connected to
Bus 504.
I/O interface 505 is connected to lower component: the importation 506 including keyboard, mouse etc.；It is penetrated including such as cathode
Spool (CRT, Cathode Ray Tube), liquid crystal display (LCD, Liquid Crystal Display) etc. and loudspeaker
Deng output par, c 507；Storage section 508 including hard disk etc.；And including such as LAN (local area network, Local Area
Network) the communications portion 509 of the network interface card of card, modem etc..Communications portion 509 is via such as internet
Network executes communication process.Driver 510 is also connected to I/O interface 505 as needed.Detachable media 511, such as disk,
CD, magneto-optic disk, semiconductor memory etc. are mounted on as needed on driver 510, in order to from the calculating read thereon
Machine program is mounted into storage section 508 as needed.
Particularly, in accordance with an embodiment of the present disclosure, it may be implemented as computer above with reference to the process of flow chart description
Software program.For example, embodiment of the disclosure includes a kind of computer program product comprising be carried on computer-readable medium
On computer program, which includes the program code for method shown in execution flow chart.In such reality
It applies in example, which can be downloaded and installed from network by communications portion 509, and/or from detachable media
511 are mounted.When the computer program is executed by central processing unit (CPU) 501, limited in execution the present processes
Above-mentioned function.It should be noted that computer-readable medium described herein can be computer-readable signal media or
Computer readable storage medium either the two any combination.Computer readable storage medium for example can be --- but
Be not limited to --- electricity, magnetic, optical, electromagnetic, infrared ray or semiconductor system, device or device, or any above combination.
The more specific example of computer readable storage medium can include but is not limited to: have one or more conducting wires electrical connection,
Portable computer diskette, hard disk, random access storage device (RAM), read-only memory (ROM), erasable type may be programmed read-only deposit
Reservoir (EPROM or flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), light storage device, magnetic memory
Part or above-mentioned any appropriate combination.In this application, computer readable storage medium, which can be, any include or stores
The tangible medium of program, the program can be commanded execution system, device or device use or in connection.And
In the application, computer-readable signal media may include in a base band or the data as the propagation of carrier wave a part are believed
Number, wherein carrying computer-readable program code.The data-signal of this propagation can take various forms, including but not
It is limited to electromagnetic signal, optical signal or above-mentioned any appropriate combination.Computer-readable signal media can also be computer
Any computer-readable medium other than readable storage medium storing program for executing, the computer-readable medium can send, propagate or transmit use
In by the use of instruction execution system, device or device or program in connection.Include on computer-readable medium
Program code can transmit with any suitable medium, including but not limited to: wireless, electric wire, optical cable, RF etc., Huo Zheshang
Any appropriate combination stated.
The calculating of the operation for executing the application can be write with one or more programming languages or combinations thereof
Machine program code, described program design language include object oriented program language-such as Java, Smalltalk, C+
+, it further include conventional procedural programming language-such as " C " language or similar programming language.Program code can
Fully to execute, partly execute on the user computer on the user computer, be executed as an independent software package,
Part executes on the remote computer or executes on a remote computer or server completely on the user computer for part.
In situations involving remote computers, remote computer can pass through the network of any kind --- including local area network (LAN)
Or wide area network (WAN)-is connected to subscriber computer, or, it may be connected to outer computer (such as utilize Internet service
Provider is connected by internet).
Flow chart and block diagram in attached drawing are illustrated according to the system of the various embodiments of the application, method and computer journey
The architecture, function and operation in the cards of sequence product.In this regard, each box in flowchart or block diagram can generation
A part of one module, program segment or code of table, a part of the module, program segment or code include one or more use
The executable instruction of the logic function as defined in realizing.It should also be noted that in some implementations as replacements, being marked in box
The function of note can also occur in a different order than that indicated in the drawings.For example, two boxes succeedingly indicated are actually
It can be basically executed in parallel, they can also be executed in the opposite order sometimes, and this depends on the function involved.Also it to infuse
Meaning, the combination of each box in block diagram and or flow chart and the box in block diagram and or flow chart can be with holding
The dedicated hardware based system of functions or operations as defined in row is realized, or can use specialized hardware and computer instruction
Combination realize.
Being described in unit involved in the embodiment of the present application can be realized by way of software, can also be by hard
The mode of part is realized.Described unit also can be set in the processor, for example, can be described as: a kind of processor packet
Include first acquisition unit, second acquisition unit and display unit.Wherein, the title of these units is not constituted under certain conditions
Restriction to the unit itself, for example, display unit is also described as " unit of acquired log is presented ".
As on the other hand, present invention also provides a kind of computer-readable medium, which be can be
Included in device described in above-described embodiment；It is also possible to individualism, and without in the supplying device.Above-mentioned calculating
Machine readable medium carries one or more program, when said one or multiple programs are executed by the device, so that should
Device: the log stored in above-mentioned log storage server is obtained；Obtain the log stored in above-mentioned O&M server；And
Acquired log is presented.
Above description is only the preferred embodiment of the application and the explanation to institute's application technology principle.Those skilled in the art
Member is it should be appreciated that invention scope involved in the application, however it is not limited to technology made of the specific combination of above-mentioned technical characteristic
Scheme, while should also cover in the case where not departing from foregoing invention design, it is carried out by above-mentioned technical characteristic or its equivalent feature
Any combination and the other technical solutions formed.Such as features described above has similar function with (but being not limited to) disclosed herein
Can technical characteristic replaced mutually and the technical solution that is formed.
Claims (11)
1. a kind of system for acquiring log, including at least one kubernetes cluster, caas cluster, log collection configuration
Server, log storage server and O&M server, kubernetes cluster include host node and at least one working node,
Kafka cluster application is installed in host node, in which:
Host node is configured to: acquiring the component log of each kubernetes component in the host node, and will be collected
Component log is sent to kafka cluster indicated by the kafka cluster application in the host node；In response to listening to the host node
Container changes in corresponding working node, according to current in the changed working node of the corresponding container of the host node
Each container state generates the log collection configuration information of relevant work node and is sent to the log collection configuration server；
Working node is configured to: obtaining the work section from the log collection configuration server every the first prefixed time interval
The log collection configuration information of point；According to acquired log collection configuration information, the kubernetes of the working node is acquired
Component log and container log, and log collected is sent to the kafka installed in the corresponding host node of the working node
Kafka cluster indicated by cluster application；
The Caas cluster is configured to: being acquired the component log of each Caas component in the Caas cluster, and will be acquired
Component log be sent to the O&M server；
The log storage server is configured to: obtaining the Caas from the Caas cluster every the second prefixed time interval
The corresponding log read of kafka cluster application installed in the host node for each kubernetes cluster that cluster is managed matches confidence
Breath；Log is pulled according to position indicated by acquired log read configuration information and is stored；
The O&M server is configured to: obtaining the log stored in the log storage server；Obtain the O&M clothes
The log stored in business device；And acquired log is presented.
2. system according to claim 1, wherein be equipped with ElasticSearch application in the O&M server；With
And
The O&M server is further configured to: being applied using the ElasticSearch installed and is carried out to acquired log
Clustering index.
3. system according to claim 2, wherein be equipped with Kibana application in the O&M server；And
The O&M server is configured to: acquired log is presented, comprising:
The O&M server is configured to: utilizing installed Kibana application that acquired log is presented.
4. a kind of method for acquiring log, applied to the O&M server in the system for acquiring log, wherein described
System for acquiring log includes at least one kubernetes cluster, caas cluster, log collection configuration server, log
Storage server and O&M server, kubernetes cluster include host node and at least one working node, are pacified in host node
Equipped with kafka cluster application, which comprises
Obtain the log stored in the log storage server；
Obtain the log stored in the O&M server；
And acquired log is presented.
5. according to the method described in claim 4, wherein, ElasticSearch application is equipped in the O&M server；With
And
The method also includes:
It is applied using the ElasticSearch installed and clustering index is carried out to acquired log.
6. according to the method described in claim 5, wherein, Kibana application is equipped in the O&M server；And
Log acquired in the presentation, comprising:
Utilize installed Kibana application that acquired log is presented.
7. it is a kind of for acquiring the device of log, applied to the O&M server in the system for acquiring log, wherein described
System for acquiring log includes at least one kubernetes cluster, caas cluster, log collection configuration server, log
Storage server and O&M server, kubernetes cluster include host node and at least one working node, are pacified in host node
Equipped with kafka cluster application, described device includes:
First acquisition unit is configured to obtain the log stored in the log storage server；
Second acquisition unit is configured to obtain the log stored in the O&M server；
Display unit is configured to present acquired log.
8. device according to claim 7, wherein be equipped with ElasticSearch application in the O&M server；With
And
Described device further include:
Indexing units are configured to be applied using the ElasticSearch installed to acquired log progress clustering index.
9. device according to claim 8, wherein be equipped with Kibana application in the O&M server；And
The display unit is further configured to:
Utilize installed Kibana application that acquired log is presented.
10. a kind of O&M server, comprising:
One or more processors；
Storage device is stored thereon with one or more programs,
When one or more of programs are executed by one or more of processors, so that one or more of processors
Realize the method as described in any in claim 4-6.
11. a kind of computer readable storage medium, is stored thereon with computer program, wherein the computer program is by one
Or multiple processors realize the method as described in any in claim 4-6 when executing.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811574616.4A CN109660397B (en) | 2018-12-21 | 2018-12-21 | System, method and device for collecting logs |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811574616.4A CN109660397B (en) | 2018-12-21 | 2018-12-21 | System, method and device for collecting logs |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109660397A true CN109660397A (en) | 2019-04-19 |
| CN109660397B CN109660397B (en) | 2022-02-15 |
Family
ID=66115966
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811574616.4A Active CN109660397B (en) | 2018-12-21 | 2018-12-21 | System, method and device for collecting logs |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109660397B (en) |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110377489A (en) * | 2019-07-17 | 2019-10-25 | 国家电网有限公司 | The method and apparatus of log information processing |
| CN110719209A (en) * | 2019-10-31 | 2020-01-21 | 北京浪潮数据技术有限公司 | Cluster network configuration method, system, equipment and readable storage medium |
| CN110955578A (en) * | 2019-11-26 | 2020-04-03 | 深圳市信联征信有限公司 | Log collection method and device based on host machine, computer equipment and storage medium |
| CN111176789A (en) * | 2019-12-30 | 2020-05-19 | 重庆紫光华山智安科技有限公司 | Container set exception handling method and device, storage medium and server |
| CN111190875A (en) * | 2019-12-27 | 2020-05-22 | 航天信息股份有限公司 | Log aggregation method and device based on container platform |
| CN111625419A (en) * | 2020-05-15 | 2020-09-04 | 浪潮电子信息产业股份有限公司 | Log acquisition method, system, equipment and computer readable storage medium |
| CN111930700A (en) * | 2020-07-13 | 2020-11-13 | 车智互联（北京）科技有限公司 | Distributed log processing method, server, system and computing equipment |
| CN112083971A (en) * | 2020-09-15 | 2020-12-15 | 北京凌云雀科技有限公司 | Component processing method, device and storage medium |
| CN112181773A (en) * | 2020-09-29 | 2021-01-05 | 银盛支付服务股份有限公司 | Method for automatically collecting module logs |
| CN112398823A (en) * | 2020-11-03 | 2021-02-23 | 内蒙古电力(集团)有限责任公司内蒙古电力科学研究院分公司 | Network information safety early warning platform based on big data analysis |
| CN112527459A (en) * | 2020-12-16 | 2021-03-19 | 新浪网技术(中国)有限公司 | Log analysis method and device based on Kubernetes cluster |
| CN112988528A (en) * | 2019-12-13 | 2021-06-18 | 北京百度网讯科技有限公司 | Log processing method and device and container group |
| CN114461490A (en) * | 2021-12-31 | 2022-05-10 | 广东航宇卫星科技有限公司 | Operation and maintenance data aggregation system |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102411533A (en) * | 2011-08-08 | 2012-04-11 | 浪潮电子信息产业股份有限公司 | Log-management optimizing method for clustered storage system |
| CN106027643A (en) * | 2016-05-18 | 2016-10-12 | 无锡华云数据技术服务有限公司 | Resource scheduling method based on Kubernetes container cluster management system |
| CN106709003A (en) * | 2016-12-23 | 2017-05-24 | 长沙理工大学 | Hadoop-based mass log data processing method |
| CN107508722A (en) * | 2017-08-01 | 2017-12-22 | 北京德塔精要信息技术有限公司 | A kind of business monitoring method and device |
| CN108521339A (en) * | 2018-03-13 | 2018-09-11 | 广州西麦科技股份有限公司 | A kind of reaction type node failure processing method and system based on cluster daily record |
| CN108921551A (en) * | 2018-06-11 | 2018-11-30 | 西安纸贵互联网科技有限公司 | Alliance's block catenary system based on Kubernetes platform |
-
2018
- 2018-12-21 CN CN201811574616.4A patent/CN109660397B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102411533A (en) * | 2011-08-08 | 2012-04-11 | 浪潮电子信息产业股份有限公司 | Log-management optimizing method for clustered storage system |
| CN106027643A (en) * | 2016-05-18 | 2016-10-12 | 无锡华云数据技术服务有限公司 | Resource scheduling method based on Kubernetes container cluster management system |
| CN106709003A (en) * | 2016-12-23 | 2017-05-24 | 长沙理工大学 | Hadoop-based mass log data processing method |
| CN107508722A (en) * | 2017-08-01 | 2017-12-22 | 北京德塔精要信息技术有限公司 | A kind of business monitoring method and device |
| CN108521339A (en) * | 2018-03-13 | 2018-09-11 | 广州西麦科技股份有限公司 | A kind of reaction type node failure processing method and system based on cluster daily record |
| CN108921551A (en) * | 2018-06-11 | 2018-11-30 | 西安纸贵互联网科技有限公司 | Alliance's block catenary system based on Kubernetes platform |
Non-Patent Citations (1)
| Title |
|---|
| 崔雅荣等: "基于Filebeat自动收集Kubernetes日志的分析系统", 《计算机系统应用》 * |
Cited By (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110377489A (en) * | 2019-07-17 | 2019-10-25 | 国家电网有限公司 | The method and apparatus of log information processing |
| CN110719209A (en) * | 2019-10-31 | 2020-01-21 | 北京浪潮数据技术有限公司 | Cluster network configuration method, system, equipment and readable storage medium |
| CN110955578A (en) * | 2019-11-26 | 2020-04-03 | 深圳市信联征信有限公司 | Log collection method and device based on host machine, computer equipment and storage medium |
| CN112988528A (en) * | 2019-12-13 | 2021-06-18 | 北京百度网讯科技有限公司 | Log processing method and device and container group |
| CN112988528B (en) * | 2019-12-13 | 2024-04-05 | 北京百度网讯科技有限公司 | Log processing method, device and container group |
| CN111190875A (en) * | 2019-12-27 | 2020-05-22 | 航天信息股份有限公司 | Log aggregation method and device based on container platform |
| CN111176789B (en) * | 2019-12-30 | 2021-08-06 | 重庆紫光华山智安科技有限公司 | Container set exception handling method and device, storage medium and server |
| CN111176789A (en) * | 2019-12-30 | 2020-05-19 | 重庆紫光华山智安科技有限公司 | Container set exception handling method and device, storage medium and server |
| CN111625419A (en) * | 2020-05-15 | 2020-09-04 | 浪潮电子信息产业股份有限公司 | Log acquisition method, system, equipment and computer readable storage medium |
| CN111930700A (en) * | 2020-07-13 | 2020-11-13 | 车智互联（北京）科技有限公司 | Distributed log processing method, server, system and computing equipment |
| CN112083971A (en) * | 2020-09-15 | 2020-12-15 | 北京凌云雀科技有限公司 | Component processing method, device and storage medium |
| CN112083971B (en) * | 2020-09-15 | 2023-04-11 | 北京凌云雀科技有限公司 | Component processing method, device and storage medium |
| CN112181773A (en) * | 2020-09-29 | 2021-01-05 | 银盛支付服务股份有限公司 | Method for automatically collecting module logs |
| CN112398823A (en) * | 2020-11-03 | 2021-02-23 | 内蒙古电力(集团)有限责任公司内蒙古电力科学研究院分公司 | Network information safety early warning platform based on big data analysis |
| CN112527459A (en) * | 2020-12-16 | 2021-03-19 | 新浪网技术(中国)有限公司 | Log analysis method and device based on Kubernetes cluster |
| CN112527459B (en) * | 2020-12-16 | 2024-03-26 | 新浪技术（中国）有限公司 | Log analysis method and device based on Kubernetes cluster |
| CN114461490A (en) * | 2021-12-31 | 2022-05-10 | 广东航宇卫星科技有限公司 | Operation and maintenance data aggregation system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109660397B (en) | 2022-02-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109660397A (en) | For acquiring system, the method and apparatus of log | |
| CN110019279B (en) | Online document collaborative updating method, device, equipment and storage medium | |
| CN110688142B (en) | Method, device and storage medium for publishing application programming interface | |
| CN103049271A (en) | Method and device for automatically generating description document of API (application program interface) | |
| CN110232054A (en) | Log transmission system and streaming log transmission method | |
| CN109213486A (en) | Method and apparatus for generating customized visualization component | |
| CN110321544B (en) | Method and device for generating information | |
| CN110391938A (en) | Method and apparatus for deployment services | |
| CN109981322A (en) | The method and apparatus of cloud resource management based on label | |
| CN110263565A (en) | Method and apparatus for calling service | |
| CN109241033A (en) | The method and apparatus for creating real-time data warehouse | |
| CN109284281A (en) | Method and apparatus for migrating data | |
| CN109783197A (en) | Dispatching method and device for program runtime environment | |
| CN110889263A (en) | Document backlog generation method, device, equipment and storage medium | |
| CN110019158A (en) | A kind of method and apparatus of monitoring data quality | |
| CN110019539A (en) | A kind of method and apparatus that the data of data warehouse are synchronous | |
| CN109033312A (en) | Method and apparatus for obtaining information | |
| CN113886485A (en) | Data processing method, device, electronic equipment, system and storage medium | |
| CN109614549B (en) | Method and apparatus for pushed information | |
| CN108885686A (en) | Active debugging system based on cloud for video analysis | |
| CN109766392A (en) | Data processing method and device | |
| CN108984564A (en) | Data-storage system, method and apparatus | |
| CN109828830A (en) | Method and apparatus for managing container | |
| CN109492039A (en) | A kind of recording method of daily record data, device, electronic equipment and readable medium | |
| CN109819042A (en) | For providing the method and apparatus of Software Development Kit |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |