CN109828960A - Log library expansion method, system, computer installation and readable storage medium storing program for executing - Google Patents
Log library expansion method, system, computer installation and readable storage medium storing program for executing Download PDFInfo
- Publication number
- CN109828960A CN109828960A CN201811537206.2A CN201811537206A CN109828960A CN 109828960 A CN109828960 A CN 109828960A CN 201811537206 A CN201811537206 A CN 201811537206A CN 109828960 A CN109828960 A CN 109828960A
- Authority
- CN
- China
- Prior art keywords
- cluster
- log
- flow
- data node
- creation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Abstract
The present invention provides a kind of log library expansion method, system, computer installation and computer readable storage medium.Log library expansion method includes: one Kubernetes cluster of creation；The current log flow of the ELK log system is obtained according to preset flow statistic algorithm；Whether the current log flow acquired described in judgement is greater than preset flow；And when the current log flow acquired is greater than preset flow, the Kubernetes cluster creation ES-Data node is called, and the ES-Data node of creation is added in the ES cluster, to increase the handling capacity of the ES cluster.The present invention is based on the log flows that cloud monitoring technology carrys out real-time monitoring log system, and carry out dilatation, dilatation service invariant to log system come dynamic according to the log uninterrupted, and high reliablity promotes user experience.
Description
Technical field
The present invention relates to daily record data process field more particularly to a kind of log library expansion methods, system, computer installation
And computer readable storage medium.
Background technique
At present most of the log system of enterprise be all based on ELK (Elasticsearch, Logstash and Kibana's
It referred to as) develops, existing ELK log system needs to be manually operated in dilatation, operationally there is very big limitation, operation is not
It is convenient, occur a series of problems after often there are many invisible factors to may result in log system dilatation during dilatation, one
Denier goes wrong, if it is desired to restore to state before, generally require the regular hour, seriously affect user experience.
Summary of the invention
In view of above-mentioned, the present invention provides a kind of log library expansion method, system, computer installation and computer-readable storage
Medium can realize automatic dilatation, dilatation service invariant.
One embodiment of the application provides a kind of log library expansion method, which comprises
A Kubernetes cluster is created, wherein ES cluster of the Kubernetes cluster independently of ELK log system；
The current log flow of the ELK log system is obtained according to preset flow statistic algorithm；
Whether the current log flow acquired described in judgement is greater than preset flow；And
When the current log flow acquired is greater than preset flow, the Kubernetes cluster is called to create
ES-Data node, and the ES-Data node of creation is added in the ES cluster, to increase the energy of handling up of the ES cluster
Power.
Preferably, the current log flow that the ELK log system is obtained according to preset flow statistic algorithm
Step includes:
The daily record data of ELK log system is obtained, and the daily record data is subjected to parsing and is converted into unified log
Format；And
It is extracted using SQL statement and passes through time sequence analysis algorithm using time, IP address as the daily record data of main fields
Statistics obtains the current log flow of the ELK log system.
Preferably, described when the current log flow acquired is greater than preset flow, described in calling
Kubernetes cluster create ES-Data node the step of include:
Every occupied physical resource of ES-Data node in the ES cluster is obtained, wherein the physical resource packet
Include cpu resource, memory source and I/O resource；And
When there are an occupied physical resources of ES-Data node more than default resource in the ES cluster and/or work as
When the current log flow acquired is greater than the preset flow, the Kubernetes cluster creation ES- is called
Data node.
Preferably, described when the current log flow acquired is greater than preset flow, described in calling
Kubernetes cluster create ES-Data node the step of include:
When the current log flow acquired be greater than the preset flow when, calculate the current log flow with
The differential flow of the preset flow；And
The Kubernetes cluster is called to create one or more ES-Data node according to the differential flow, wherein
The number of the ES-Data node of the creation is matched with the size of the differential flow.
Preferably, the step that the ES-Data node of creation is added in the ES cluster includes:
Zen discovery mechanism is established in the ES cluster, and passes through the Zen discovery mechanism for the Kubernetes collection
The ES-Data node of group's creation is added in the ES cluster.
Preferably, described that the ES-Data node of creation is added in the ES cluster, to increase gulping down for the ES cluster
Further include after the step of spitting ability；
Whether the current log flow for monitoring the ELK log system is less than the preset flow；And
When the current log flow of the ELK log system is less than the preset flow, by the Kubernetes collection
Data on the ES-Data node of group's creation are synchronized to the ES cluster, and after completing data and synchronizing described in release
The ES-Data node of Kubernetes cluster creation.
Preferably, described that the ES-Data node of creation is added in the ES cluster, to increase gulping down for the ES cluster
After the step of spitting ability further include:
The current log flow for monitoring the ELK log system whether is less than the preset flow and whether the duration is big
In a preset time；And
When the current log flow of the ELK log system be less than the preset flow and duration be greater than it is described default
When the time, the data on the ES-Data node of Kubernetes cluster creation are synchronized to the ES cluster, and complete
Data discharge the ES-Data node of the Kubernetes cluster creation after synchronizing.
One embodiment of the application provides a kind of log library capacity-enlarging system, the system comprises:
Creation module, for creating a Kubernetes cluster, wherein the Kubernetes cluster is independently of ELK log
The ES cluster of system；
Module is obtained, for obtaining the current log flow of the ELK log system according to preset flow statistic algorithm；
Judgment module, for judging whether the current log flow acquired is greater than preset flow；And
Dilatation module, for when the current log flow acquired is greater than preset flow, described in calling
Kubernetes cluster creates ES-Data node, and the ES-Data node of creation is added in the ES cluster, to increase
The handling capacity of the ES cluster.
One embodiment of the application provides a kind of computer installation, and the computer installation includes processor and memory,
Several computer programs are stored on the memory, the processor is for when executing the computer program stored in memory
The step of realizing log library expansion method as elucidated before.
One embodiment of the application provides a kind of computer readable storage medium, is stored thereon with computer program, described
The step of log library expansion method as elucidated before is realized when computer program is executed by processor.
Above-mentioned log library expansion method, system, computer installation and computer readable storage medium, by creation independently of
The Kubernetes cluster of ES cluster, to realize according to the size of current log flow dynamically dilatation ES cluster, once detecting
When being more than preset flow to flow, Kubernetes is called to create ES-Data node, thus increase the handling capacity of ES cluster,
After flow restores normal, the data on the ES-Data node of Kubernetes dynamic creation out can also be synchronized to ES collection
Group, and the ES-Data node that Kubernetes dynamic creation goes out is discharged, so that ELK log system can also restore to original automatically
State, dilatation service invariant, high reliablity promote user experience.
Detailed description of the invention
It, below will be to required in embodiment description in order to illustrate more clearly of the technical solution of embodiment of the present invention
The attached drawing used is briefly described, it should be apparent that, the accompanying drawings in the following description is some embodiments of the present invention, for
For those of ordinary skill in the art, without creative efforts, it can also be obtained according to these attached drawings other
Attached drawing.
Fig. 1 is the step flow chart of log library expansion method in one embodiment of the invention.
Fig. 2 is the step flow chart of log library expansion method in another embodiment of the present invention.
Fig. 3 is the functional block diagram of log library capacity-enlarging system in one embodiment of the invention.
Fig. 4 is computer schematic device in one embodiment of the invention.
Specific embodiment
To better understand the objects, features and advantages of the present invention, with reference to the accompanying drawing and specific real
Applying mode, the present invention will be described in detail.It should be noted that in the absence of conflict, presently filed embodiment and reality
The feature applied in mode can be combined with each other.
In the following description, numerous specific details are set forth in order to facilitate a full understanding of the present invention, described embodiment
Only some embodiments of the invention, rather than whole embodiments.Based on the embodiment in the present invention, this field
Those of ordinary skill's every other embodiment obtained without making creative work, belongs to guarantor of the present invention
The range of shield.
Unless otherwise defined, all technical and scientific terms used herein and belong to technical field of the invention
The normally understood meaning of technical staff is identical.Term as used herein in the specification of the present invention is intended merely to description tool
The purpose of the embodiment of body, it is not intended that in the limitation present invention.
Preferably, log library expansion method of the invention is applied in one or more computer installation.The calculating
Machine device is that one kind can be according to the instruction for being previously set or storing, the automatic equipment for carrying out numerical value calculating and/or information processing,
Its hardware includes but is not limited to microprocessor, specific integrated circuit (Application Specific Integrated
Circuit, ASIC), programmable gate array (Field-Programmable Gate Array, FPGA), digital processing unit
(Digital Signal Processor, DSP), embedded device etc..
The computer installation can be the calculating such as desktop PC, laptop, tablet computer, server and set
It is standby.The computer installation can carry out people by modes such as keyboard, mouse, remote controler, touch tablet or voice-operated devices with user
Machine interaction.
Embodiment one:
Fig. 1 is the step flow chart of log library expansion method preferred embodiment of the present invention.The stream according to different requirements,
The sequence of step can change in journey figure, and certain steps can be omitted.
As shown in fig.1, log library expansion method specifically includes following steps.
Step S11, a Kubernetes cluster is created, wherein the Kubernetes cluster is independently of ELK log system
ES cluster.
It in one embodiment, include ES cluster, Logstash frame and Kibana frame in the ELK log system
Frame.The Logstash frame can be distributed on each node of ES cluster, can be used for collecting correlation log data, and through excessive
ES cluster is sent to after analysis, filtering to be stored, then is presented to the user daily record data by Kibana frame, for example provide a variety of
API is for user query, operation.
In one embodiment, Kubernetes cluster can be created in ELK log system, create Kubernetes collection
The mode of group can be realized by following steps: setting create cloud virtual machine quantity used in the Kubernetes cluster and
Specification, and complete the creation of required cloud virtual machine；Obtain the IP information and ssh (Secure Shell, Secure Shell of cloud virtual machine
Agreement) information；Binary file required for deployment Kubernetes cluster is copied to the cloud created using ssh tool
On virtual machine, and Kubernetes collection swarm parameter is set；Finally using all of kubectl tool deployment Kubernetes cluster
Component.The Kubernetes cluster is mutually decoupled with the ES cluster in the ELK log system, i.e., the described Kubernetes collection
Group, the ES cluster are mutually indepedent.
Step S12, the current log flow of the ELK log system is obtained according to preset flow statistic algorithm.
In one embodiment, the current log stream of the ELK log system is counted according to preset flow statistic algorithm
The mode of amount specifically may is that the daily record data for obtaining ELK log system, can be in order to reduce follow-up data processing workload
Only acquisition current time counts forward the daily record data of a preset time, such as obtains the log number in first three minute at current time
According to；The daily record data for recycling shell script, python script or Apache script will acquire carries out parsing and is converted into unification
Journal format；It is finally extracted using SQL statement and passes through time series using time, IP address as the daily record data of main fields
Parser counts to obtain the current log flow of the ELK log system.
In one embodiment, can also be can be with real-time statistics ELK log system number for the preset flow statistic algorithm
It according to the algorithm of pack receiving and transmitting information, is received and sent messages by the data packet of ELK log system described in real-time statistics, and then available institute
State the current log uninterrupted of ELK log system.
Step S13, whether the current log flow acquired described in judgement is greater than preset flow.
In one embodiment, it can be carried out by the current log flow and preset flow for acquiring step S12
Compare, to judge whether the current log flow is greater than preset flow.The preset flow can refer to the ELK days aspiration
The relevant information of system is set, such as runs bandwidth, the operation ELK log that the ELK log system equipment is accessed
The software-hardware configuration information of system equipment, profile information of the ELK log system itself etc..
Step S14, when the current log flow acquired is greater than preset flow, the Kubernetes is called
Cluster creates ES-Data node, and the ES-Data node of creation is added in the ES cluster, to increase the ES cluster
Handling capacity.
In one embodiment, the ES cluster itself includes multiple nodes, those nodes can be divided into ES-
Master node, ES-Data node and ES-Client node.Wherein ES-Master node is mainly used for being responsible for management cluster
The change of scope, such as creation or index is deleted, addition node is to cluster or from cluster deletion of node.ES-Data node is main
It is the node for storing index data, document progress additions and deletions is changed and look into operation, converging operation etc..ES-Client node is main
It is for playing the part of a load balancing when ES-Master nodal community and ES-Data nodal community both are set to false
The request of arrival is routed to each node in cluster by role.
In one embodiment, ES cluster can be divided into a complete index multiple fragments, and by multiple fragment
It is distributed on different nodes, after the ES-Data node of Kubernetes cluster creation is added to the ES cluster, institute
The fragment can be redistributed by stating ES cluster, and then may be implemented to configure newly added ES-Data node.
When the current log flow acquired is greater than preset flow, the Kubernetes cluster is called to create
ES-Data node, and the ES-Data node of creation is added in the ES cluster, and then the ES cluster can be increased
Handling capacity.Specifically, Zen discovery mechanism can be established in the ES cluster, the Zen discovery mechanism, which may be implemented to work as, to be had
When one new ES node is added and title identical with the ES cluster is arranged, this new node can be added into ES cluster
In, and then can be realized by the Zen discovery mechanism and be added to the ES-Data node that the Kubernetes cluster creates
In the ES cluster.It should be understood that being not necessarily to when the current log flow acquired is not more than the preset flow
Call the Kubernetes cluster creation ES-Data node.
In one embodiment, it calls the quantity of the ES-Data node of the Kubernetes cluster creation and works as the day before yesterday
Will flow is related to the differential flow of the preset flow, when the differential flow is bigger, the quantity of the ES-Data node of creation
It is more.When the current log flow acquired be greater than the preset flow when, also calculate the current log flow with
The differential flow of the preset flow, and the Kubernetes cluster creation one is called according to the differential flow being calculated
A or multiple ES-Data nodes, wherein the number of the ES-Data node of the creation is matched with the size of the differential flow.
In one embodiment, it is occupied can also to obtain every ES-Data node in the ES cluster by step S12
Physical resource, and then can be according to every occupied physical resource of ES-Data node in the ES cluster in step S14
It whether is more than default resource to determine whether calling the Kubernetes cluster creation ES-Data node.Specifically, institute is obtained
Every occupied physical resource of ES-Data node in ES cluster is stated, wherein the physical resource includes cpu resource, memory
Resource and I/O resource；When there are an occupied physical resource of ES-Data node being more than default resource in the ES cluster or
When the current log flow acquired is greater than the preset flow, the Kubernetes cluster creation ES- is called
Data node.
It is in one embodiment, acceptable that only there are the occupied physics moneys of an ES-Data node in the ES cluster
When source is more than default resource and when the current log flow acquired is greater than the preset flow, described in calling
Kubernetes cluster creates ES-Data node.
In one embodiment, in step s 12, it can also count ELK days described by preset flow statistic algorithm
Aspiration is united in the log flow of a designated time period, such as the log flow in half a year, and is built according to the log flow of acquisition
The log flow of the ELK log system is predicted in vertical log flux prediction model realization, predicts log flow
Tendency, obtains the log traffic prediction value of next timing node, and whether the log flow that step S13 may determine that prediction obtains is big
In preset threshold, to determine the need for creating an ES-Data node using the Kubernetes cluster in advance.Implement one
In mode, the log flux prediction model can be established using Kalman filtering algorithm.
Please refer to Fig. 2, compared with the expansion method of log library shown in fig. 1, Fig. 2 shows log library expansion method also
Including step S15 and step S16.
Whether step S15, the current log flow for monitoring the ELK log system are less than the preset flow；
Step S16 will be described when the current log flow of the ELK log system is less than the preset flow
Data on the ES-Data node of Kubernetes cluster creation are synchronized to the ES cluster, and release after completing data and synchronizing
Put the ES-Data node of the Kubernetes cluster creation.
In one embodiment, when the current log flow for monitoring the ELK log system is less than the preset flow,
Show that the own node of the ES cluster meets ELK log system demand enough, without additional node, can incite somebody to action at this time
Additional increased ES-Data node exits the ES cluster.Additional increased ES-Data node is being exited into the ES collection simultaneously
Before group, the data information by the ES-Data node is needed to be synchronized to the ES cluster, avoids that loss of data occurs.As the ELK
When the current log flow of log system is less than the preset flow, the ES-Data of Kubernetes cluster creation is saved
Data on point are synchronized to the ES cluster, and the ES- of the Kubernetes cluster creation is discharged after completing data and synchronizing
Data node, and then realize and dynamic capacity-expanding is carried out to ELK log system, when being not necessarily to dilatation, ELK log system can also be automatic extensive
Again to original state.It, can be with when the current log flow for monitoring the ELK log system again is greater than the preset flow
The Kubernetes cluster creation ES-Data node is called, again to be added to the ES cluster.
In one embodiment, in order to avoid repeat release/creation ES-Data node, preferably monitor it is ELK days described
After the log flow of aspiration system is less than the preset flow and continues a preset time, then the ES-Data node that will additionally create
Data information be synchronized to the ES cluster, to discharge the ES-Data node that those are additionally created.The preset time can root
It is set according to actual demand, for example, the preset time is set as 3 minutes.
Above-mentioned log library expansion method is by creating the Kubernetes cluster independently of ES cluster, to realize according to current
Dynamically dilatation ES cluster calls Kubernetes wound when detecting flow more than preset flow to the size of log flow
ES-Data node is built, to increase the handling capacity of ES cluster, after flow restores normal, Kubernetes can also be moved
The data on ES-Data node that state is created that are synchronized to ES cluster, and discharge the ES-Data that Kubernetes dynamic creation goes out
Node promotes user and uses so that ELK log system can also restore automatically to original state, dilatation service invariant, high reliablity
Experience.
Embodiment two:
Fig. 3 is the functional block diagram of log library capacity-enlarging system preferred embodiment of the present invention.
As shown in fig.3, log library capacity-enlarging system 10 may include creation module 101, obtain module 102, judgement
Module 103, dilatation module 104, monitoring modular 105 and release module 106.
The creation module 101 for create a Kubernetes cluster, wherein the Kubernetes cluster independently of
The ES cluster of ELK log system.
It in one embodiment, include ES cluster, Logstash frame and Kibana frame in the ELK log system
Frame.The Logstash frame can be distributed on each node of ES cluster, can be used for collecting correlation log data, and through excessive
ES cluster is sent to after analysis, filtering to be stored, then is presented to the user daily record data by Kibana frame, for example provide a variety of
API is for user query, operation.
In one embodiment, the creation module 101 can create Kubernetes cluster in ELK log system,
The mode of specific creation Kubernetes cluster can be realized by following steps: setting creates the Kubernetes cluster institute
The cloud virtual machine quantity and specification used, and complete the creation of required cloud virtual machine；Obtain the IP information and ssh of cloud virtual machine
(Secure Shell, safety shell protocol) information；Using ssh tool copy deployment Kubernetes cluster required for two into
On file processed to the cloud virtual machine created, and Kubernetes collection swarm parameter is set；Finally use kubectl tool part
Affix one's name to all components of Kubernetes cluster.The Kubernetes cluster and the ES cluster in the ELK log system are mutual
Decoupling, i.e., the described Kubernetes cluster, the ES cluster are mutually indepedent.
It is described obtain that module 102 is used to obtain the ELK log system according to preset flow statistic algorithm work as the day before yesterday
Will flow.
In one embodiment, the acquisition module 102 can obtain the ELK log system in the following manner
Current log flow: obtaining the daily record data of ELK log system, in order to reduce follow-up data processing workload, can only obtain
Current time counts forward the daily record data of a preset time, such as obtains the daily record data in first three minute at current time；It is sharp again
Parsing is carried out with the daily record data that shell script, python script or Apache script will acquire and is converted into unified log
Format；It is finally extracted using SQL statement using time, IP address as the daily record data of main fields, and is calculated by time series analysis
Method counts to obtain the current log flow of the ELK log system.
In one embodiment, can also be can be with real-time statistics ELK log system number for the preset flow statistic algorithm
It according to the algorithm of pack receiving and transmitting information, is received and sent messages by the data packet of ELK log system described in real-time statistics, and then available institute
State the current log uninterrupted of ELK log system.
The judgment module 103 is used to judge whether the current log flow acquired to be greater than preset flow.
In one embodiment, the current log that the judgment module 103 can be acquired by will acquire module 102
Flow is compared with preset flow, to judge whether the current log flow is greater than preset flow.The preset flow can
It is set with the relevant information with reference to the ELK log system, such as runs the band that the ELK log system equipment is accessed
The profile information of wide, the operation ELK log system equipment software-hardware configuration information, the ELK log system itself
Deng.
The dilatation module 104 is used to call institute when the current log flow acquired is greater than preset flow
Kubernetes cluster creation ES-Data node is stated, and the ES-Data node of creation is added in the ES cluster, to increase
Add the handling capacity of the ES cluster.
In one embodiment, the ES cluster itself includes multiple nodes, those nodes can be divided into ES-
Master node, ES-Data node and ES-Client node.Wherein ES-Master node is mainly used for being responsible for management cluster
The change of scope, such as creation or index is deleted, addition node is to cluster or from cluster deletion of node.ES-Data node is main
It is the node for storing index data, document progress additions and deletions is changed and look into operation, converging operation etc..ES-Client node is main
It is for playing the part of a load balancing when ES-Master nodal community and ES-Data nodal community both are set to false
The request of arrival is routed to each node in cluster by role.
In one embodiment, ES cluster can be divided into a complete index multiple fragments, and by multiple fragment
It is distributed on different nodes, after the ES-Data node of Kubernetes cluster creation is added to the ES cluster, institute
The fragment can be redistributed by stating ES cluster, and then may be implemented to configure newly added ES-Data node.
When the current log flow acquired is greater than preset flow, the dilatation module 104 is called described
Kubernetes cluster creates ES-Data node, and the ES-Data node of creation is added in the ES cluster, Jin Erke
To increase the handling capacity of the ES cluster.The dilatation module 104 can establish Zen discovery mechanism in the ES cluster,
The Zen discovery mechanism may be implemented when there is a new ES node to be added and title identical with the ES cluster is arranged,
This new node can be added into ES cluster, and then can be realized by the Zen discovery mechanism will be described
The ES-Data node of Kubernetes cluster creation is added in the ES cluster.It should be understood that acquired described in
When current log flow is not more than the preset flow, without calling the Kubernetes cluster creation ES-Data node.
In one embodiment, the dilatation module 104 calls the ES-Data node of the Kubernetes cluster creation
Quantity it is related with the differential flow of the preset flow to current log flow, when the differential flow is bigger, the ES- of creation
The quantity of Data node is more.When the current log flow acquired is greater than the preset flow, the dilatation mould
Block 104 also calculates the differential flow of the current log flow and the preset flow, and according to the differential flow being calculated
Call Kubernetes cluster creation one or more ES-Data node, wherein the ES-Data node of the creation
Number is matched with the size of the differential flow.
In one embodiment, the acquisition module 102 can also obtain every ES-Data node in the ES cluster
Occupied physical resource, and then the dilatation module 104 can be according to shared by every ES-Data node in the ES cluster
Whether physical resource is more than default resource to determine whether calling the Kubernetes cluster creation ES-Data node.
Specifically, every occupied physical resource of ES-Data node in the ES cluster is obtained, the acquisition module 102 is wherein
The physical resource includes cpu resource, memory source and I/O resource；When there are shared by an ES-Data node in the ES cluster
When physical resource is more than default resource or when the current log flow acquired is greater than the preset flow, institute
It states dilatation module 104 and calls the Kubernetes cluster creation ES-Data node.
In one embodiment, the dilatation module 104 can also only there are an ES-Data nodes in the ES cluster
When occupied physical resource is more than default resource and when the current log flow acquired is greater than the preset flow,
Call the Kubernetes cluster creation ES-Data node.
In one embodiment, the acquisition module 102 can also count to obtain institute by preset flow statistic algorithm
ELK log system is stated in the log flow of a designated time period, such as the log flow in half a year, and according to the log stream of acquisition
Amount predicts the log flow of the ELK log system to establish log flux prediction model realization, predicts log
The tendency of flow, obtains the log traffic prediction value of next timing node, and the judgment module 103 may determine that prediction obtained
Whether log flow is greater than preset threshold, and the dilatation module 104 can be to determine according to the judging result of judgment module 103
It is no to need to create an ES-Data node using the Kubernetes cluster in advance.In one embodiment, the log flow
Prediction model can be established using Kalman filtering algorithm.
Whether the current log flow that the monitoring modular 105 is used to monitor the ELK log system is less than described default
Flow.
The release module 106 is used for when the current log flow of the ELK log system is less than the preset flow,
Data on the ES-Data node of Kubernetes cluster creation are synchronized to the ES cluster, and same completing data
The ES-Data node of the Kubernetes cluster creation is discharged after step.
In one embodiment, when the current log flow that the monitoring modular 105 monitors the ELK log system is less than
When the preset flow, show that the own node of the ES cluster meets ELK log system demand enough, without additional
Node, additional increased ES-Data node can be exited the ES cluster by the release module 106 at this time.Simultaneously by volume
Before outer increased ES-Data node exits the ES cluster, the data information by the ES-Data node is needed to be synchronized to the ES
Cluster avoids that loss of data occurs.I.e. when the current log flow of the ELK log system is less than the preset flow, institute
It states release module 106 and the data on ES-Data node that the Kubernetes cluster creates is synchronized to the ES cluster, and
The ES-Data node of the Kubernetes cluster creation is discharged after completing data and synchronizing, and then is realized to ELK log system
Dynamic capacity-expanding is carried out, when being not necessarily to dilatation, ELK log system can also restore to original state automatically.When monitoring modular 105 again
When monitoring that the current log flow of the ELK log system is greater than the preset flow, the dilatation module 104 can be again
The Kubernetes cluster creation ES-Data node is called, to be added to the ES cluster.
In one embodiment, in order to avoid repeating release/creation ES-Data node, the release module 106 preferably exists
After monitoring that the log flow of the ELK log system is less than the preset flow and continues a preset time, then will additionally it create
The data information for the ES-Data node built is synchronized to the ES cluster, to discharge the ES-Data node that those are additionally created.Institute
Stating preset time can be set according to actual needs, for example, the preset time is set as 3 minutes.
Above-mentioned log library capacity-enlarging system is by creating the Kubernetes cluster independently of ES cluster, to realize according to current
Dynamically dilatation ES cluster calls Kubernetes wound when detecting flow more than preset flow to the size of log flow
ES-Data node is built, to increase the handling capacity of ES cluster, after flow restores normal, Kubernetes can also be moved
The data on ES-Data node that state is created that are synchronized to ES cluster, and discharge the ES-Data that Kubernetes dynamic creation goes out
Node promotes user and uses so that ELK log system can also restore automatically to original state, dilatation service invariant, high reliablity
Experience.
Fig. 4 is the schematic diagram of computer installation preferred embodiment of the present invention.
The computer installation 1 includes memory 20, processor 30 and is stored in the memory 20 and can be in institute
State the computer program 40 run on processor 30, such as log library dilatation program.The processor 30 executes the computer
The step in above-mentioned log library expansion method embodiment, such as step S11~S14 shown in FIG. 1, Fig. 2 institute are realized when program 40
Step S11~the S16 shown.Alternatively, the processor 30 realizes above-mentioned log library dilatation system when executing the computer program 40
The function of each module in embodiment of uniting, such as the module 101~106 in Fig. 3.
Illustratively, the computer program 40 can be divided into one or more module/units, it is one or
Multiple module/units are stored in the memory 20, and are executed by the processor 30, to complete the present invention.Described one
A or multiple module/units can be the series of computation machine program instruction section that can complete specific function, and described instruction section is used
In implementation procedure of the description computer program 40 in the computer installation 1.For example, the computer program 40 can be with
It is divided into the creation module 101 in Fig. 3, obtains module 102, judgment module 103, dilatation module 104, monitoring modular 105, releases
Amplification module 106.Each module concrete function is referring to embodiment two.
The computer installation 1 can be the calculating such as desktop PC, notebook, palm PC and cloud server and set
It is standby.It will be understood by those skilled in the art that the schematic diagram is only the example of computer installation 1, do not constitute to computer
The restriction of device 1 may include perhaps combining certain components or different components, example than illustrating more or fewer components
Such as described computer installation 1 can also include input-output equipment, network access equipment, bus.
Alleged processor 30 can be central processing unit (Central Processing Unit, CPU), can also be
Other general processors, digital signal processor (Digital Signal Processor, DSP), specific integrated circuit
(Application Specific Integrated Circuit, ASIC), ready-made programmable gate array (Field-
Programmable Gate Array, FPGA) either other programmable logic device, discrete gate or transistor logic,
Discrete hardware components etc..General processor can be microprocessor or the processor 30 is also possible to any conventional processing
Device etc., the processor 30 are the control centres of the computer installation 1, utilize various interfaces and the entire computer of connection
The various pieces of device 1.
The memory 20 can be used for storing the computer program 40 and/or module/unit, and the processor 30 passes through
Operation executes the computer program and/or module/unit being stored in the memory 20, and calls and be stored in memory
Data in 20 realize the various functions of the computer installation 1.The memory 20 can mainly include storing program area and deposit
Store up data field, wherein storing program area can application program needed for storage program area, at least one function (for example sound is broadcast
Playing function, image player function etc.) etc.；Storage data area, which can be stored, uses created data (ratio according to computer installation 1
Such as audio data, phone directory) etc..In addition, memory 20 may include high-speed random access memory, it can also include non-easy
The property lost memory, such as hard disk, memory, plug-in type hard disk, intelligent memory card (Smart Media Card, SMC), secure digital
(Secure Digital, SD) card, flash card (Flash Card), at least one disk memory, flush memory device or other
Volatile solid-state part.
If the integrated module/unit of the computer installation 1 is realized in the form of SFU software functional unit and as independence
Product when selling or using, can store in a computer readable storage medium.Based on this understanding, of the invention
It realizes all or part of the process in above-described embodiment method, can also instruct relevant hardware come complete by computer program
At the computer program can be stored in a computer readable storage medium, and the computer program is held by processor
When row, it can be achieved that the step of above-mentioned each embodiment of the method.Wherein, the computer program includes computer program code, institute
Stating computer program code can be source code form, object identification code form, executable file or certain intermediate forms etc..It is described
Computer-readable medium may include: any entity or device, recording medium, U that can carry the computer program code
Disk, mobile hard disk, magnetic disk, CD, computer storage, read-only memory (ROM, Read-Only Memory), arbitrary access
Memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium etc..It needs
It is bright, the content that the computer-readable medium includes can according in jurisdiction make laws and patent practice requirement into
Row increase and decrease appropriate, such as do not include electric load according to legislation and patent practice, computer-readable medium in certain jurisdictions
Wave signal and telecommunication signal.
In several embodiments provided by the present invention, it should be understood that disclosed computer installation and method, it can be with
It realizes by another way.For example, computer installation embodiment described above is only schematical, for example, described
The division of unit, only a kind of logical function partition, there may be another division manner in actual implementation.
It, can also be in addition, each functional unit in each embodiment of the present invention can integrate in same treatment unit
It is that each unit physically exists alone, can also be integrated in same unit with two or more units.Above-mentioned integrated list
Member both can take the form of hardware realization, can also realize in the form of hardware adds software function module.
It is obvious to a person skilled in the art that invention is not limited to the details of the above exemplary embodiments, Er Qie
In the case where without departing substantially from spirit or essential attributes of the invention, the present invention can be realized in other specific forms.Therefore, no matter
From the point of view of which point, the present embodiments are to be considered as illustrative and not restrictive, and the scope of the present invention is by appended power
Benefit requires rather than above description limits, it is intended that all by what is fallen within the meaning and scope of the equivalent elements of the claims
Variation is included in the present invention.Any reference signs in the claims should not be construed as limiting the involved claims.This
Outside, it is clear that one word of " comprising " does not exclude other units or steps, and odd number is not excluded for plural number.It is stated in computer installation claim
Multiple units or computer installation can also be implemented through software or hardware by the same unit or computer installation.The
One, the second equal words are used to indicate names, and are not indicated any particular order.
Finally it should be noted that the above examples are only used to illustrate the technical scheme of the present invention and are not limiting, although reference
Preferred embodiment describes the invention in detail, those skilled in the art should understand that, it can be to of the invention
Technical solution is modified or equivalent replacement, without departing from the spirit and scope of the technical solution of the present invention.
Claims (10)
1. a kind of log library expansion method, which is characterized in that the described method includes:
A Kubernetes cluster is created, wherein ES cluster of the Kubernetes cluster independently of ELK log system；
The current log flow of the ELK log system is obtained according to preset flow statistic algorithm；
Whether the current log flow acquired described in judgement is greater than preset flow；And
When the current log flow acquired is greater than preset flow, the Kubernetes cluster creation ES- is called
Data node, and the ES-Data node of creation is added in the ES cluster, to increase the handling capacity of the ES cluster.
2. log library as described in claim 1 expansion method, which is characterized in that described to be obtained according to preset flow statistic algorithm
The step of taking the current log flow of the ELK log system include:
The daily record data of ELK log system is obtained, and the daily record data is subjected to parsing and is converted into unified journal format；
And
It is extracted using SQL statement using time, IP address as the daily record data of main fields, and is counted by time sequence analysis algorithm
Obtain the current log flow of the ELK log system.
3. log library as described in claim 1 expansion method, which is characterized in that described when the current log acquired
When flow is greater than preset flow, the step of calling Kubernetes cluster creation ES-Data node, includes:
Every occupied physical resource of ES-Data node in the ES cluster is obtained, wherein the physical resource includes CPU
Resource, memory source and I/O resource；And
When there are an occupied physical resources of ES-Data node more than default resource in the ES cluster and/or when described
When the current log flow acquired is greater than the preset flow, the Kubernetes cluster creation ES-Data section is called
Point.
4. log library as described in claim 1 expansion method, which is characterized in that described when the current log acquired
When flow is greater than preset flow, the step of calling Kubernetes cluster creation ES-Data node, includes:
When the current log flow acquired be greater than the preset flow when, calculate the current log flow with it is described
The differential flow of preset flow；And
The Kubernetes cluster is called to create one or more ES-Data node according to the differential flow, wherein described
The number of the ES-Data node of creation is matched with the size of the differential flow.
5. the log library expansion method as described in claim 1-4 any one, which is characterized in that the ES- by creation
The step that Data node is added in the ES cluster includes:
Zen discovery mechanism is established in the ES cluster, and is created the Kubernetes cluster by the Zen discovery mechanism
The ES-Data node built is added in the ES cluster.
6. log library as claimed in claim 1 or 2 expansion method, which is characterized in that described to add the ES-Data node of creation
Add in the ES cluster, the step of handling capacity to increase the ES cluster after further include；
Whether the current log flow for monitoring the ELK log system is less than the preset flow；And
When the current log flow of the ELK log system is less than the preset flow, the Kubernetes cluster is created
The data on ES-Data node built are synchronized to the ES cluster, and discharge the Kubernetes after completing data and synchronizing
The ES-Data node of cluster creation.
7. log library as claimed in claim 1 or 2 expansion method, which is characterized in that described to add the ES-Data node of creation
Add in the ES cluster, the step of handling capacity to increase the ES cluster after further include:
The current log flow for monitoring the ELK log system whether is less than the preset flow and whether the duration is greater than one
Preset time；And
It is greater than the preset time when the current log flow of the ELK log system is less than the preset flow and duration
When, the data on the ES-Data node of Kubernetes cluster creation are synchronized to the ES cluster, and complete data
The ES-Data node of the Kubernetes cluster creation is discharged after synchronizing.
8. a kind of log library capacity-enlarging system, which is characterized in that the system comprises:
Creation module, for creating a Kubernetes cluster, wherein the Kubernetes cluster is independently of ELK log system
ES cluster；
Module is obtained, for obtaining the current log flow of the ELK log system according to preset flow statistic algorithm；
Judgment module, for judging whether the current log flow acquired is greater than preset flow；And
Dilatation module, for when the current log flow acquired is greater than preset flow, described in calling
Kubernetes cluster creates ES-Data node, and the ES-Data node of creation is added in the ES cluster, to increase
The handling capacity of the ES cluster.
9. a kind of computer installation, the computer installation includes processor and memory, is stored on the memory several
Computer program, which is characterized in that such as right is realized when the processor is for executing the computer program stored in memory
It is required that described in any one of 1-7 the step of the expansion method of log library.
10. a kind of computer readable storage medium, is stored thereon with computer program, which is characterized in that the computer program
The step of log library expansion method as described in any one of claim 1-7 is realized when being executed by processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811537206.2A CN109828960A (en) | 2018-12-14 | 2018-12-14 | Log library expansion method, system, computer installation and readable storage medium storing program for executing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811537206.2A CN109828960A (en) | 2018-12-14 | 2018-12-14 | Log library expansion method, system, computer installation and readable storage medium storing program for executing |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109828960A true CN109828960A (en) | 2019-05-31 |
Family
ID=66858885
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811537206.2A Pending CN109828960A (en) | 2018-12-14 | 2018-12-14 | Log library expansion method, system, computer installation and readable storage medium storing program for executing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109828960A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110287163A (en) * | 2019-06-25 | 2019-09-27 | 浙江乾冠信息安全研究院有限公司 | Security log acquires analytic method, device, equipment and medium |
| CN111752489A (en) * | 2020-06-30 | 2020-10-09 | 重庆紫光华山智安科技有限公司 | Expansion method of PVC (polyvinyl chloride) module in Kubernetes and related device |
| CN111953551A (en) * | 2020-08-27 | 2020-11-17 | 网易（杭州）网络有限公司 | Log data transmission method and device, electronic equipment and storage medium |
| CN112714018A (en) * | 2020-12-28 | 2021-04-27 | 上海领健信息技术有限公司 | Gateway-based ElasticSearch search service method, system, medium and terminal |
| CN113225362A (en) * | 2020-02-06 | 2021-08-06 | 北京京东振世信息技术有限公司 | Server cluster system and implementation method thereof |
| CN114584346A (en) * | 2022-01-28 | 2022-06-03 | 深圳融安网络科技有限公司 | Log stream processing method, system, terminal device and storage medium |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101840314A (en) * | 2010-05-05 | 2010-09-22 | 北京星网锐捷网络技术有限公司 | Method, device and server for expanding storage space of database |
| CN102968503A (en) * | 2012-12-10 | 2013-03-13 | 曙光信息产业(北京)有限公司 | Data processing method for database system, and database system |
| US20150100549A1 (en) * | 2013-10-03 | 2015-04-09 | International Business Machines Corporation | Extending a content repository using an auxiliary data store |
| CN105095417A (en) * | 2015-07-13 | 2015-11-25 | 小米科技有限责任公司 | Method and apparatus for storing business data and database node |
| CN105138615A (en) * | 2015-08-10 | 2015-12-09 | 北京思特奇信息技术股份有限公司 | Method and system for building big data distributed log |
| CN106547471A (en) * | 2015-09-17 | 2017-03-29 | 北京国双科技有限公司 | The extended method and device of non-relational database |
| CN107357896A (en) * | 2017-07-13 | 2017-11-17 | 北京小度信息科技有限公司 | Expansion method, device, system and the data base cluster system of data-base cluster |
| CN107862090A (en) * | 2017-12-04 | 2018-03-30 | 中企动力科技股份有限公司 | The horizontal extension method and device of database |
| CN108282522A (en) * | 2018-01-15 | 2018-07-13 | 吉浦斯信息咨询（深圳）有限公司 | Data storage access method based on dynamic routing and system |
| CN108667654A (en) * | 2018-04-19 | 2018-10-16 | 北京奇艺世纪科技有限公司 | The automatic expansion method of server cluster and relevant device |
| CN108712296A (en) * | 2018-06-07 | 2018-10-26 | 郑州云海信息技术有限公司 | One kind being based on distributed daily record monitoring device and method |
-
2018
- 2018-12-14 CN CN201811537206.2A patent/CN109828960A/en active Pending
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101840314A (en) * | 2010-05-05 | 2010-09-22 | 北京星网锐捷网络技术有限公司 | Method, device and server for expanding storage space of database |
| CN102968503A (en) * | 2012-12-10 | 2013-03-13 | 曙光信息产业(北京)有限公司 | Data processing method for database system, and database system |
| US20150100549A1 (en) * | 2013-10-03 | 2015-04-09 | International Business Machines Corporation | Extending a content repository using an auxiliary data store |
| CN105095417A (en) * | 2015-07-13 | 2015-11-25 | 小米科技有限责任公司 | Method and apparatus for storing business data and database node |
| CN105138615A (en) * | 2015-08-10 | 2015-12-09 | 北京思特奇信息技术股份有限公司 | Method and system for building big data distributed log |
| CN106547471A (en) * | 2015-09-17 | 2017-03-29 | 北京国双科技有限公司 | The extended method and device of non-relational database |
| CN107357896A (en) * | 2017-07-13 | 2017-11-17 | 北京小度信息科技有限公司 | Expansion method, device, system and the data base cluster system of data-base cluster |
| CN107862090A (en) * | 2017-12-04 | 2018-03-30 | 中企动力科技股份有限公司 | The horizontal extension method and device of database |
| CN108282522A (en) * | 2018-01-15 | 2018-07-13 | 吉浦斯信息咨询（深圳）有限公司 | Data storage access method based on dynamic routing and system |
| CN108667654A (en) * | 2018-04-19 | 2018-10-16 | 北京奇艺世纪科技有限公司 | The automatic expansion method of server cluster and relevant device |
| CN108712296A (en) * | 2018-06-07 | 2018-10-26 | 郑州云海信息技术有限公司 | One kind being based on distributed daily record monitoring device and method |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110287163A (en) * | 2019-06-25 | 2019-09-27 | 浙江乾冠信息安全研究院有限公司 | Security log acquires analytic method, device, equipment and medium |
| CN110287163B (en) * | 2019-06-25 | 2021-10-08 | 浙江乾冠信息安全研究院有限公司 | Method, device, equipment and medium for collecting and analyzing security log |
| CN113225362A (en) * | 2020-02-06 | 2021-08-06 | 北京京东振世信息技术有限公司 | Server cluster system and implementation method thereof |
| CN113225362B (en) * | 2020-02-06 | 2024-04-05 | 北京京东振世信息技术有限公司 | Server cluster system and implementation method thereof |
| CN111752489A (en) * | 2020-06-30 | 2020-10-09 | 重庆紫光华山智安科技有限公司 | Expansion method of PVC (polyvinyl chloride) module in Kubernetes and related device |
| CN111752489B (en) * | 2020-06-30 | 2022-06-17 | 重庆紫光华山智安科技有限公司 | Expansion method of PVC (polyvinyl chloride) module in Kubernetes and related device |
| CN111953551A (en) * | 2020-08-27 | 2020-11-17 | 网易（杭州）网络有限公司 | Log data transmission method and device, electronic equipment and storage medium |
| CN112714018A (en) * | 2020-12-28 | 2021-04-27 | 上海领健信息技术有限公司 | Gateway-based ElasticSearch search service method, system, medium and terminal |
| CN114584346A (en) * | 2022-01-28 | 2022-06-03 | 深圳融安网络科技有限公司 | Log stream processing method, system, terminal device and storage medium |
| CN114584346B (en) * | 2022-01-28 | 2024-01-12 | 深圳融安网络科技有限公司 | Log stream processing method, system, terminal equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109828960A (en) | Log library expansion method, system, computer installation and readable storage medium storing program for executing | |
| CN108776934B (en) | Distributed data calculation method and device, computer equipment and readable storage medium | |
| CN110535831A (en) | Cluster safety management method, device and storage medium based on Kubernetes and network domains | |
| KR101871383B1 (en) | Method and system for using a recursive event listener on a node in hierarchical data structure | |
| CN110096683A (en) | Report form generation method, system, computer installation and computer readable storage medium | |
| CN105630589A (en) | Distributed process scheduling system and process scheduling and execution method | |
| CN109729147A (en) | The auditing system and implementation method of multi-tenant are supported under a kind of cloud environment | |
| CN112839067B (en) | Data synchronization method and device | |
| CN109840142A (en) | Thread control method, device, electronic equipment and storage medium based on cloud monitoring | |
| CN110262902A (en) | Information processing method and system, medium and calculating equipment | |
| CN109451078A (en) | Transaction methods and device under a kind of distributed structure/architecture | |
| CN110796341B (en) | Distributed batch processing method and device for collecting cases and electronic equipment | |
| CN109302324A (en) | A kind of private clound monitoring and early warning method and system | |
| CN109348434A (en) | A kind of sending method of scene information, sending device and terminal device | |
| CN114222086A (en) | Method, system, medium and electronic device for scheduling audio and video code stream | |
| CN113783913A (en) | Message pushing management method and device | |
| CN102769495B (en) | A kind of optical fiber access network equipment communication means, Apparatus and system | |
| CN113778565A (en) | Interface data processing method and device | |
| CN110321252B (en) | Skill service resource scheduling method and device | |
| CN111614702B (en) | Edge calculation method and edge calculation system | |
| CN115629909A (en) | Service data processing method and device, electronic equipment and storage medium | |
| CN114756301A (en) | Log processing method, device and system | |
| CN113794755A (en) | Shared service pushing method and system based on micro-service architecture | |
| CN108696549B (en) | Load balancing method, device and system | |
| CN113297516A (en) | Customer interaction interface generation method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |