CN109828960B

CN109828960B - Log library capacity expansion method, system, computer device and readable storage medium

Info

Publication number: CN109828960B
Application number: CN201811537206.2A
Authority: CN
Inventors: 石晓龙; 饶鑫; 黄望
Original assignee: Ping An Technology Shenzhen Co Ltd
Current assignee: Ping An Technology Shenzhen Co Ltd
Priority date: 2018-12-14
Filing date: 2018-12-14
Publication date: 2024-05-28
Anticipated expiration: 2038-12-14
Also published as: CN109828960A

Abstract

The invention provides a log library capacity expansion method, a log library capacity expansion system, a computer device and a computer readable storage medium. The log library capacity expansion method comprises the following steps: creating a Kubernetes cluster; acquiring the current log flow of the ELK log system according to a preset flow statistics algorithm; judging whether the obtained current log flow is larger than a preset flow or not; and when the obtained current log flow is greater than a preset flow, calling the Kubernetes cluster to create an ES-Data node, and adding the created ES-Data node into the ES cluster to increase the throughput capacity of the ES cluster. According to the invention, the log flow of the log system is monitored in real time based on the cloud monitoring technology, the capacity of the log system is dynamically expanded according to the log flow, the capacity expansion service is stable, the reliability is high, and the user experience is improved.

Description

Log library capacity expansion method, system, computer device and readable storage medium

Technical Field

The present invention relates to the field of log data processing, and in particular, to a log library capacity expansion method, system, computer device and computer readable storage medium.

Background

Most of the log systems of the current enterprises are developed based on ELK (ELASTICSEARCH, LOGSTASH and Kibana for short), the existing ELK log systems need to be manually operated during capacity expansion, have great limitation on operation, are inconvenient to operate, often have a plurality of invisible factors during capacity expansion, can cause a series of problems after the capacity expansion of the log systems, and once the problems occur, if the log systems want to recover to the previous state, a certain time is needed, and the user experience is seriously affected.

Disclosure of Invention

In view of the foregoing, the present invention provides a log library capacity expansion method, system, computer device and computer readable storage medium, which can implement automatic capacity expansion and stable capacity expansion service.

The embodiment of the application provides a log library capacity expansion method, which comprises the following steps:

creating a Kubernetes cluster, wherein the Kubernetes cluster is independent of an ES cluster of an ELK log system;

acquiring the current log flow of the ELK log system according to a preset flow statistics algorithm;

judging whether the obtained current log flow is larger than a preset flow or not; and

And when the obtained current log flow is greater than a preset flow, calling the Kubernetes cluster to create an ES-Data node, and adding the created ES-Data node into the ES cluster to increase the throughput capacity of the ES cluster.

Preferably, the step of obtaining the current log traffic of the ELK log system according to a preset traffic statistics algorithm includes:

Acquiring log data of an ELK log system, and analyzing and converting the log data into a unified log format; and

And extracting log data taking time and an IP address as main fields by using SQL sentences, and carrying out statistics by using a time sequence analysis algorithm to obtain the current log flow of the ELK log system.

Preferably, the step of calling the Kubernetes cluster to create the ES-Data node when the obtained current log traffic is greater than a preset traffic includes:

Acquiring physical resources occupied by each ES-Data node in the ES cluster, wherein the physical resources comprise CPU resources, memory resources and I/O resources; and

And when the physical resource occupied by one ES-Data node in the ES cluster exceeds a preset resource and/or when the current acquired log flow is greater than the preset flow, invoking the Kubernetes cluster to create the ES-Data node.

when the obtained current log flow is larger than the preset flow, calculating the difference flow between the current log flow and the preset flow; and

And calling the Kubernetes cluster to create one or more ES-Data nodes according to the difference flow, wherein the number of the created ES-Data nodes is matched with the size of the difference flow.

Preferably, the step of adding the created ES-Data node to the ES cluster comprises:

And establishing a Zen discovery mechanism in the ES cluster, and adding the ES-Data node created by the Kubernetes cluster into the ES cluster through the Zen discovery mechanism.

Preferably, the step of adding the created ES-Data node to the ES cluster to increase throughput capacity of the ES cluster further comprises;

monitoring whether the current log flow of the ELK log system is smaller than the preset flow; and

And synchronizing Data on the ES-Data node created by the Kubernetes cluster to the ES cluster when the current log flow of the ELK log system is smaller than the preset flow, and releasing the ES-Data node created by the Kubernetes cluster after the Data synchronization is completed.

Preferably, the step of adding the created ES-Data node to the ES cluster to increase throughput capacity of the ES cluster further comprises:

monitoring whether the current log flow of the ELK log system is smaller than the preset flow and whether the duration time is longer than a preset time; and

And synchronizing Data on the ES-Data node created by the Kubernetes cluster to the ES cluster when the current log flow of the ELK log system is smaller than the preset flow and the duration is longer than the preset time, and releasing the ES-Data node created by the Kubernetes cluster after the Data synchronization is completed.

An embodiment of the present application provides a log library capacity expansion system, which includes:

a creation module for creating a Kubernetes cluster, wherein the Kubernetes cluster is independent of an ES cluster of an ELK log system;

The acquisition module is used for acquiring the current log flow of the ELK log system according to a preset flow statistics algorithm;

the judging module is used for judging whether the obtained current log flow is larger than a preset flow or not; and

And the capacity expansion module is used for calling the Kubernetes cluster to create an ES-Data node when the obtained current log flow is larger than a preset flow, and adding the created ES-Data node into the ES cluster to increase the throughput capacity of the ES cluster.

An embodiment of the present application provides a computer device, where the computer device includes a processor and a memory, where the memory stores a plurality of computer programs, and the processor is configured to implement the steps of the log library capacity expansion method when executing the computer programs stored in the memory.

An embodiment of the present application provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the log library capacity expansion method as described above.

According to the log library capacity expansion method, the system, the computer device and the computer readable storage medium, the Kubernetes cluster independent of the ES cluster is created, so that the dynamic capacity expansion of the ES cluster according to the current log flow is realized, once the current log flow is detected to exceed the preset flow, the Kubernetes is called to create the ES-Data node, the throughput capacity of the ES cluster is increased, after the flow is recovered to be normal, the Data on the ES-Data node dynamically created by the Kubernetes can be synchronized to the ES cluster, and the ES-Data node dynamically created by the Kubernetes is released, so that the ELK log system can be automatically restored to the original state, the capacity expansion service is stable, the reliability is high, and the user experience is improved.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a flowchart illustrating a method for log library expansion according to an embodiment of the present invention.

FIG. 2 is a flowchart illustrating a method for log library expansion according to another embodiment of the present invention.

FIG. 3 is a functional block diagram of a log library capacity expansion system according to an embodiment of the present invention.

FIG. 4 is a schematic diagram of a computer device according to an embodiment of the invention.

Detailed Description

In order that the above-recited objects, features and advantages of the present application will be more clearly understood, a more particular description of the application will be rendered by reference to the appended drawings and appended detailed description. The embodiments of the present application and the features in the embodiments may be combined with each other without collision.

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, and the described embodiments are merely some, rather than all, embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, are intended to fall within the scope of the present invention.

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

Preferably, the log library capacity expansion method of the present invention is applied in one or more computer devices. The computer apparatus is a device capable of automatically performing numerical calculation and/or information processing according to a preset or stored instruction, and its hardware includes, but is not limited to, a microprocessor, an Application SPECIFIC INTEGRATED Circuit (ASIC), a Programmable gate array (Field-Programmable GATE ARRAY, FPGA), a digital Processor (DIGITAL SIGNAL Processor, DSP), an embedded device, and the like.

The computer device may be a computing device such as a desktop computer, a notebook computer, a tablet computer, a server, etc. The computer device can perform man-machine interaction with a user through a keyboard, a mouse, a remote controller, a touch pad or voice control equipment and the like.

Embodiment one:

FIG. 1 is a flowchart illustrating steps of a log library capacity expansion method according to a preferred embodiment of the present invention. The order of the steps in the flow diagrams may be changed, and some steps may be omitted, according to different needs.

Referring to fig. 1, the log library capacity expansion method specifically includes the following steps.

Step S11, creating a Kubernetes cluster, wherein the Kubernetes cluster is independent of an ES cluster of the ELK log system.

In one embodiment, the ELK log system includes an ES cluster, a logstack framework, and a Kibana framework. The Logstar framework can be distributed on each node of the ES cluster, can be used for collecting related log data, and is sent to the ES cluster for storage after analysis and filtration, and then the Kibana framework presents the log data to a user, for example, provides various APIs for the user to inquire and operate.

In an embodiment, a Kubernetes cluster may be created in an ELK log system, and the manner in which the Kubernetes cluster is created may be achieved by: setting the number and specification of cloud virtual machines used for creating the Kubernetes cluster, and completing the creation of the required cloud virtual machines; acquiring IP information and ssh (Secure Shell, secure Shell protocol) information of a cloud virtual machine; copying binary files required by deploying the Kubernetes cluster to the created cloud virtual machine by using a ssh tool, and setting the parameters of the Kubernetes cluster; finally all components of the Kubernetes cluster are deployed using kubectl tools. The Kubernetes clusters are decoupled from the ES clusters in the ELK log system, i.e., the Kubernetes clusters and the ES clusters are independent from each other.

And step S12, acquiring the current log flow of the ELK log system according to a preset flow statistics algorithm.

In an embodiment, the method for counting the current log traffic of the ELK log system according to the preset traffic statistics algorithm may specifically be: the log data of the ELK log system may be obtained, in order to reduce the workload of subsequent data processing, only the log data of a preset time counted forward at the current time, for example, the log data in three minutes before the current time may be obtained; then analyzing and converting the acquired log data into a unified log format by utilizing a shell script, a python script or an Apache script; and finally, extracting log data taking time and an IP address as main fields by utilizing SQL sentences, and carrying out statistics by a time sequence analysis algorithm to obtain the current log flow of the ELK log system.

In an embodiment, the preset flow statistics algorithm may be an algorithm capable of counting data packet transceiving information of the ELK log system in real time, and further obtaining the current log flow of the ELK log system by counting the data packet transceiving information of the ELK log system in real time.

And S13, judging whether the obtained current log flow is larger than a preset flow.

In an embodiment, whether the current log flow is greater than the preset flow may be determined by comparing the current log flow obtained in step S12 with the preset flow. The preset flow may be set with reference to related information of the ELK log system, for example, a bandwidth accessed by running the ELK log system device, software and hardware configuration information of running the ELK log system device, configuration file information of the ELK log system itself, and the like.

And S14, when the obtained current log flow is greater than a preset flow, calling the Kubernetes cluster to create an ES-Data node, and adding the created ES-Data node into the ES cluster to increase the throughput capacity of the ES cluster.

In one embodiment, the ES cluster itself includes a plurality of nodes that may be divided into ES-Master, ES-Data, and ES-Client nodes. Wherein the ES-Master node is mainly responsible for managing changes to the cluster category, such as creating or deleting indexes, adding nodes to the cluster, or deleting nodes from the cluster. The ES-Data node is mainly used for storing index Data, and performs operations such as adding, deleting, checking, aggregating and the like on the document. The ES-Client node is primarily used to play a role of load balancing when the ES-Master node attribute and the ES-Data node attribute are both set to false, routing incoming requests to the various nodes in the cluster.

In an embodiment, an ES cluster may divide a complete index into a plurality of slices, and distribute the slices to different nodes, where after an ES-Data node created by the Kubernetes cluster is added to the ES cluster, the ES cluster redistributes the slices, so as to implement configuration of a newly added ES-Data node.

And when the obtained current log flow is greater than a preset flow, calling the Kubernetes cluster to create an ES-Data node, and adding the created ES-Data node into the ES cluster, so that the throughput capacity of the ES cluster can be increased. Specifically, a Zen discovery mechanism may be established in the ES cluster, where the Zen discovery mechanism may implement that when a new ES node joins and sets the same name as the ES cluster, the new node is joined to the ES cluster, and further may implement that the ES-Data node created by the Kubernetes cluster is added to the ES cluster through the Zen discovery mechanism. It can be appreciated that when the obtained current log traffic is not greater than the preset traffic, the Kubernetes cluster does not need to be called to create an ES-Data node.

In one embodiment, the number of ES-Data nodes created by invoking the Kubernetes cluster is related to the difference traffic between the current log traffic and the preset traffic, and the larger the difference traffic is, the larger the number of ES-Data nodes created is. And when the obtained current log flow is larger than the preset flow, calculating the difference flow between the current log flow and the preset flow, and calling the Kubernetes cluster to create one or more ES-Data nodes according to the calculated difference flow, wherein the number of the created ES-Data nodes is matched with the size of the difference flow.

In an embodiment, step S12 may further obtain the physical resources occupied by each ES-Data node in the ES cluster, and further in step S14, it may determine whether to call the Kubernetes cluster to create the ES-Data node according to whether the physical resources occupied by each ES-Data node in the ES cluster exceed the preset resources. Specifically, physical resources occupied by each ES-Data node in the ES cluster are acquired, wherein the physical resources comprise CPU resources, memory resources and I/O resources; and when the physical resource occupied by one ES-Data node in the ES cluster exceeds a preset resource or the obtained current log flow is larger than the preset flow, invoking the Kubernetes cluster to create the ES-Data node.

In an embodiment, the Kubernetes cluster may be invoked to create the ES-Data node only when the physical resource occupied by the ES-Data node in the ES cluster exceeds a preset resource and the obtained current log flow is greater than the preset flow.

In an embodiment, in step S12, the log flow of the ELK log system in a specified period of time, for example, the log flow in half a year, may be counted by a preset flow statistics algorithm, and a log flow prediction model is established according to the obtained log flow to predict the log flow of the ELK log system, so as to predict the trend of the log flow and obtain the log flow predicted value of the next time node, and step S13 may determine whether the predicted log flow is greater than a preset threshold, so as to determine whether an ES-Data node needs to be created by using the Kubernetes cluster in advance. In one embodiment, the log traffic prediction model may be built using a Kalman filtering algorithm.

Referring to fig. 2, compared with the log library capacity expansion method shown in fig. 1, the log library capacity expansion method shown in fig. 2 further includes step S15 and step S16.

Step S15, monitoring whether the current log flow of the ELK log system is smaller than the preset flow;

And S16, synchronizing Data on the ES-Data node created by the Kubernetes cluster to the ES cluster when the current log flow of the ELK log system is smaller than the preset flow, and releasing the ES-Data node created by the Kubernetes cluster after the Data synchronization is completed.

In an embodiment, when the current log traffic of the ELK log system is monitored to be smaller than the preset traffic, it indicates that the own node of the ES cluster is enough to meet the requirement of the ELK log system, and no additional node is needed, at this time, the additional added ES-Data node may be withdrawn from the ES cluster. And meanwhile, before the additionally added ES-Data node exits the ES cluster, the Data information of the ES-Data node needs to be synchronized to the ES cluster, so that the Data loss is avoided. And when the current log flow of the ELK log system is smaller than the preset flow, synchronizing the Data on the ES-Data node created by the Kubernetes cluster to the ES cluster, and releasing the ES-Data node created by the Kubernetes cluster after the Data synchronization is completed, so that dynamic capacity expansion of the ELK log system is realized, and when capacity expansion is not needed, the ELK log system can be automatically restored to the original state. When the current log flow of the ELK log system is monitored to be larger than the preset flow again, the Kubernetes cluster can be called again to create an ES-Data node so as to be added to the ES cluster.

In one embodiment, to avoid repeated releasing/creating of ES-Data nodes, it is preferable to synchronize Data information of the additionally created ES-Data nodes to the ES cluster after monitoring that the log traffic of the ELK log system is smaller than the preset traffic for a preset time to release the additionally created ES-Data nodes. The preset time may be set according to actual requirements, for example, the preset time is set to 3 minutes.

According to the log library capacity expansion method, the Kubernetes cluster independent of the ES cluster is created, so that the ES cluster is dynamically expanded according to the current log flow, once the fact that the flow exceeds the preset flow is detected, the Kubernetes is called to create the ES-Data node, so that the throughput capacity of the ES cluster is improved, after the flow is recovered to be normal, the Data on the ES-Data node dynamically created by the Kubernetes can be synchronized to the ES cluster, the ES-Data node dynamically created by the Kubernetes is released, the ELK log system can be automatically recovered to the original state, capacity expansion service is stable, reliability is high, and user experience is improved.

Embodiment two:

FIG. 3 is a functional block diagram of a log library capacity expansion system according to a preferred embodiment of the present invention.

Referring to fig. 3, the log library capacity expansion system 10 may include a creation module 101, an acquisition module 102, a judgment module 103, a capacity expansion module 104, a monitoring module 105, and a release module 106.

The creation module 101 is configured to create a Kubernetes cluster, wherein the Kubernetes cluster is independent of an ES cluster of an ELK log system.

In an embodiment, the creating module 101 may create a Kubernetes cluster in the ELK log system, and specifically, the manner of creating the Kubernetes cluster may be implemented by the following steps: setting the number and specification of cloud virtual machines used for creating the Kubernetes cluster, and completing the creation of the required cloud virtual machines; acquiring IP information and ssh (Secure Shell, secure Shell protocol) information of a cloud virtual machine; copying binary files required by deploying the Kubernetes cluster to the created cloud virtual machine by using a ssh tool, and setting the parameters of the Kubernetes cluster; finally all components of the Kubernetes cluster are deployed using kubectl tools. The Kubernetes clusters are decoupled from the ES clusters in the ELK log system, i.e., the Kubernetes clusters and the ES clusters are independent from each other.

The obtaining module 102 is configured to obtain a current log flow of the ELK log system according to a preset flow statistics algorithm.

In one embodiment, the obtaining module 102 may obtain the current log traffic of the ELK log system by: the log data of the ELK log system may be obtained, in order to reduce the workload of subsequent data processing, only the log data of a preset time counted forward at the current time, for example, the log data in three minutes before the current time may be obtained; then analyzing and converting the acquired log data into a unified log format by utilizing a shell script, a python script or an Apache script; and finally, extracting log data taking time and an IP address as main fields by utilizing SQL sentences, and carrying out statistics by a time sequence analysis algorithm to obtain the current log flow of the ELK log system.

The judging module 103 is configured to judge whether the obtained current log flow is greater than a preset flow.

In an embodiment, the determining module 103 may determine whether the current log flow is greater than a preset flow by comparing the current log flow obtained by the obtaining module 102 with the preset flow. The preset flow may be set with reference to related information of the ELK log system, for example, a bandwidth accessed by running the ELK log system device, software and hardware configuration information of running the ELK log system device, configuration file information of the ELK log system itself, and the like.

The capacity expansion module 104 is configured to call the Kubernetes cluster to create an ES-Data node when the obtained current log flow is greater than a preset flow, and add the created ES-Data node to the ES cluster, so as to increase throughput capacity of the ES cluster.

When the obtained current log flow is greater than a preset flow, the capacity expansion module 104 calls the Kubernetes cluster to create an ES-Data node, and adds the created ES-Data node to the ES cluster, so that throughput capacity of the ES cluster can be increased. The capacity expansion module 104 may establish a Zen discovery mechanism in the ES cluster, where the Zen discovery mechanism may implement that when a new ES node is added and sets the same name as the ES cluster, the new node is added to the ES cluster, and further may implement that the ES-Data node created by the Kubernetes cluster is added to the ES cluster through the Zen discovery mechanism. It can be appreciated that when the obtained current log traffic is not greater than the preset traffic, the Kubernetes cluster does not need to be called to create an ES-Data node.

In an embodiment, the capacity expansion module 104 invokes that the number of ES-Data nodes created by the Kubernetes cluster is related to a difference flow between the current log flow and the preset flow, and when the difference flow is larger, the number of created ES-Data nodes is larger. When the obtained current log traffic is greater than the preset traffic, the capacity expansion module 104 further calculates a difference traffic between the current log traffic and the preset traffic, and calls the Kubernetes cluster to create one or more ES-Data nodes according to the calculated difference traffic, wherein the number of the created ES-Data nodes is matched with the size of the difference traffic.

In an embodiment, the obtaining module 102 may further obtain physical resources occupied by each ES-Data node in the ES cluster, and the capacity expansion module 104 may determine whether to call the Kubernetes cluster to create the ES-Data node according to whether the physical resources occupied by each ES-Data node in the ES cluster exceed preset resources. Specifically, the physical resources occupied by each ES-Data node in the ES cluster are acquired, where the physical resources include CPU resources, memory resources, and I/O resources in the acquisition module 102; when the physical resource occupied by an ES-Data node in the ES cluster exceeds a preset resource or when the obtained current log flow is greater than the preset flow, the capacity expansion module 104 calls the Kubernetes cluster to create the ES-Data node.

In an embodiment, the capacity expansion module 104 may further call the Kubernetes cluster to create the ES-Data node only when the physical resource occupied by the ES-Data node in the ES cluster exceeds a preset resource and the obtained current log traffic is greater than the preset traffic.

In an embodiment, the obtaining module 102 may further calculate, through a preset flow statistics algorithm, a log flow of the ELK log system in a specified period of time, for example, a log flow within half a year, and establish a log flow prediction model according to the obtained log flow to implement prediction of the log flow of the ELK log system, predict a trend of the log flow, obtain a log flow predicted value of a next time node, where the determining module 103 may determine whether the predicted log flow is greater than a preset threshold, and the capacity expansion module 104 may determine, according to a determination result of the determining module 103, whether an ES-Data node needs to be created by using the Kubernetes cluster in advance. In one embodiment, the log traffic prediction model may be built using a Kalman filtering algorithm.

The monitoring module 105 is configured to monitor whether a current log flow of the ELK log system is less than the preset flow.

The releasing module 106 is configured to synchronize Data on an ES-Data node created by the Kubernetes cluster to the ES cluster when the current log traffic of the ELK log system is less than the preset traffic, and release the ES-Data node created by the Kubernetes cluster after the Data synchronization is completed.

In an embodiment, when the monitoring module 105 monitors that the current log traffic of the ELK log system is less than the preset traffic, it indicates that the own node of the ES cluster is enough to meet the ELK log system requirement, without using an additional node, and the releasing module 106 may exit the ES cluster from the additional ES-Data node. And meanwhile, before the additionally added ES-Data node exits the ES cluster, the Data information of the ES-Data node needs to be synchronized to the ES cluster, so that the Data loss is avoided. That is, when the current log flow of the ELK log system is smaller than the preset flow, the releasing module 106 synchronizes the Data on the ES-Data node created by the Kubernetes cluster to the ES cluster, and releases the ES-Data node created by the Kubernetes cluster after the Data synchronization is completed, so as to realize dynamic capacity expansion of the ELK log system, and when the capacity expansion is not required, the ELK log system can automatically recover to the original state. When the monitoring module 105 again monitors that the current log traffic of the ELK log system is greater than the preset traffic, the capacity expansion module 104 may call the Kubernetes cluster again to create an ES-Data node to add to the ES cluster.

In one embodiment, to avoid repeated releasing/creating of the ES-Data nodes, the releasing module 106 preferably synchronizes the Data information of the additionally created ES-Data nodes to the ES cluster to release the additionally created ES-Data nodes after monitoring that the log traffic of the ELK log system is smaller than the preset traffic for a preset time. The preset time may be set according to actual requirements, for example, the preset time is set to 3 minutes.

According to the log library capacity expansion system, the Kubernetes cluster independent of the ES cluster is created, so that the ES cluster is dynamically expanded according to the current log flow, once the fact that the flow exceeds the preset flow is detected, the Kubernetes is called to create the ES-Data node, so that the throughput capacity of the ES cluster is improved, after the flow is recovered to be normal, the Data on the ES-Data node dynamically created by the Kubernetes can be synchronized to the ES cluster, the ES-Data node dynamically created by the Kubernetes is released, the ELK log system can be automatically recovered to the original state, the capacity expansion service is stable, the reliability is high, and the user experience is improved.

FIG. 4 is a schematic diagram of a computer device according to a preferred embodiment of the invention.

The computer device 1 comprises a memory 20, a processor 30 and a computer program 40, such as a log library expansion program, stored in the memory 20 and executable on the processor 30. The processor 30 executes the computer program 40 to implement the steps in the embodiment of the log library capacity expansion method, for example, steps S11 to S14 shown in fig. 1 and steps S11 to S16 shown in fig. 2. Or the processor 30 may perform the functions of the modules in the embodiment of the log library expansion system described above, such as modules 101-106 in fig. 3, when executing the computer program 40.

Illustratively, the computer program 40 may be partitioned into one or more modules/units that are stored in the memory 20 and executed by the processor 30 to complete the present invention. The one or more modules/units may be a series of computer program instruction segments capable of performing the specified functions, which instruction segments are used for describing the execution of the computer program 40 in the computer device 1. For example, the computer program 40 may be divided into a creation module 101, an acquisition module 102, a judgment module 103, a capacity expansion module 104, a monitoring module 105, and a release module 106 in fig. 3. For specific functions of each module, see embodiment two.

The computer device 1 may be a computing device such as a desktop computer, a notebook computer, a palm computer, a cloud server, etc. It will be appreciated by a person skilled in the art that the schematic diagram is only an example of the computer apparatus 1 and does not constitute a limitation of the computer apparatus 1, and may comprise more or less components than shown, or may combine certain components, or different components, e.g. the computer apparatus 1 may further comprise input and output devices, network access devices, buses, etc.

The Processor 30 may be a central processing unit (Central Processing Unit, CPU), other general purpose Processor, digital signal Processor (DIGITAL SIGNAL Processor, DSP), application SPECIFIC INTEGRATED Circuit (ASIC), off-the-shelf Programmable gate array (Field-Programmable GATE ARRAY, FPGA) or other Programmable logic device, discrete gate or transistor logic device, discrete hardware components, or the like. The general purpose processor may be a microprocessor or the processor 30 may be any conventional processor or the like, the processor 30 being the control center of the computer device 1, the various interfaces and lines being used to connect the various parts of the overall computer device 1.

The memory 20 may be used to store the computer program 40 and/or modules/units, and the processor 30 may perform various functions of the computer device 1 by executing or executing the computer program and/or modules/units stored in the memory 20, and invoking data stored in the memory 20. The memory 20 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program (such as a sound playing function, an image playing function, etc.) required for at least one function, and the like; the storage data area may store data created according to the use of the computer apparatus 1 (such as audio data, phonebook, etc.), and the like. In addition, the memory 20 may include high-speed random access memory, and may also include non-volatile memory, such as a hard disk, memory, plug-in hard disk, smart memory card (SMART MEDIA CARD, SMC), secure Digital (SD) card, flash memory card (FLASH CARD), at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage device.

The modules/units integrated in the computer device 1 may be stored in a computer readable storage medium if implemented in the form of software functional units and sold or used as separate products. Based on such understanding, the present invention may implement all or part of the flow of the method of the above embodiment, or may be implemented by a computer program to instruct related hardware, where the computer program may be stored in a computer readable storage medium, where the computer program, when executed by a processor, may implement the steps of each of the method embodiments described above. Wherein the computer program comprises computer program code which may be in source code form, object code form, executable file or some intermediate form etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), an electrical carrier signal, a telecommunications signal, a software distribution medium, and so forth. It should be noted that the computer readable medium contains content that can be appropriately scaled according to the requirements of jurisdictions in which such content is subject to legislation and patent practice, such as in certain jurisdictions in which such content is subject to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

In the several embodiments provided by the present invention, it should be understood that the disclosed computer apparatus and method may be implemented in other ways. For example, the above-described embodiments of the computer apparatus are merely illustrative, and for example, the division of the units is merely a logical function division, and there may be other manners of division when actually implemented.

In addition, each functional unit in the embodiments of the present invention may be integrated in the same processing unit, or each unit may exist alone physically, or two or more units may be integrated in the same unit. The integrated units can be realized in a form of hardware or a form of hardware and a form of software functional modules.

It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned. Furthermore, it is evident that the word "comprising" does not exclude other elements or steps, and that the singular does not exclude a plurality. Multiple units or computer means recited in the computer means claim may also be implemented by means of software or hardware by means of the same unit or computer means. The terms first, second, etc. are used to denote a name, but not any particular order.

Finally, it should be noted that the above-mentioned embodiments are merely for illustrating the technical solution of the present invention and not for limiting the same, and although the present invention has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications and equivalents may be made to the technical solution of the present invention without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for expanding a log library, the method comprising:

Acquiring current log flow of the ELK log system according to a preset flow statistics algorithm, and predicting predicted log flow of a next time node after the current log flow based on a preset log flow prediction model, wherein the preset log flow prediction model is obtained based on log flow data training of the ELK log system in a specified time period;

Judging whether the current log flow and the predicted log flow are larger than a preset flow or not; and

And when the current log flow and/or the predicted log flow is greater than a preset flow, calling the Kubernetes cluster to create an ES-Data node, and adding the created ES-Data node into the ES cluster to increase the throughput capacity of the ES cluster.

2. The method for expanding log library according to claim 1, wherein the step of obtaining the current log traffic of the ELK log system according to a preset traffic statistics algorithm comprises:

3. The method for expanding log library according to claim 1, wherein the step of calling the Kubernetes cluster to create ES-Data node when the current log traffic and/or the predicted log traffic is greater than a preset traffic comprises:

And when the physical resource occupied by one ES-Data node in the ES cluster exceeds a preset resource and/or at least one of the current log flow and the predicted log flow is greater than the preset flow, invoking the Kubernetes cluster to create the ES-Data node.

4. The method for expanding log library according to claim 1, wherein the step of calling the Kubernetes cluster to create ES-Data node when the current log traffic and/or the predicted log traffic is greater than a preset traffic comprises:

when the current log flow and/or the predicted log flow are/is larger than the preset flow, calculating a difference flow of the larger of the current log flow and the predicted log flow and the preset flow; and

5. The log library capacity expansion method according to any one of claims 1 to 4, wherein the step of adding the created ES-Data node to the ES cluster comprises:

6. The log library capacity expansion method according to claim 1 or 2, wherein the step of adding the created ES-Data node to the ES cluster to increase throughput of the ES cluster further comprises, after:

7. A log library expansion system, the system comprising:

the acquisition module is used for acquiring the current log flow of the ELK log system according to a preset flow statistics algorithm and predicting the predicted log flow of a next time node after the current log flow based on a preset log flow prediction model, wherein the preset log flow prediction model is obtained based on log flow data training of the ELK log system in a specified time period;

the judging module is used for judging whether the current log flow and the predicted log flow are larger than a preset flow or not; and

And the capacity expansion module is used for calling the Kubernetes cluster to create an ES-Data node when the current log flow and/or the predicted log flow are/is larger than a preset flow, and adding the created ES-Data node into the ES cluster so as to increase the throughput capacity of the ES cluster.

8. A computer device comprising a processor and a memory, the memory having a number of computer programs stored thereon, wherein the processor is configured to implement the steps of the log library expansion method according to any of claims 1 to 6 when executing the computer programs stored in the memory.

9. A computer readable storage medium having stored thereon a computer program, wherein the computer program when executed by a processor implements the steps of the log library expansion method of any of claims 1 to 6.