CN106649461A - Method for automatically cleaning and maintaining elastic search log index file - Google Patents

Abstract

The invention particularly relates to a method for automatically cleaning and maintaining an ElasticSearch log index file. According to the method for automatically cleaning and maintaining the ElasticSearch log index file, the index file is stored separately according to the time dimension, a log index deleting strategy is made according to the service requirement and becomes a scheduling task, the log deleting task is scheduled by using a scheduling frame, when the historical data index is required to be deleted, the index which meets the strategy is deleted integrally according to the log index deleting strategy, and the problem of efficiency of deleting according to a DeleteByquery mode can be solved. The method for automatically cleaning and maintaining the ElasticSearch log index file can quickly and efficiently delete the index file, cannot cause performance influence on current index and query, and solves the problem that the ElasticSearch has low efficiency when a DeleteByquery mode is adopted to delete a large data volume index.

Description

A method for automatically cleaning and maintaining ElasticSearch log index files

technical field

The invention relates to the technical field of big data, in particular to a method for automatically cleaning and maintaining ElasticSearch log index files.

Background technique

In information technology, big data refers to the inability to capture, manage, store, search, and share its content within a certain period of time with conventional software tools (such as existing database management tools or data processing applications). Large and complex data collections consisting of huge quantities, complex structures, and many types of data that are processed, analyzed, and visualized. Big data has four characteristics, namely high volume (Volume), rapidity (Velocity), diversity (Variety) and low value density (Value). The challenge brought by big data lies in its real-time processing, and the data itself has changed from structured data to unstructured data, so it is very difficult to use relational databases to process big data.

Big data is often used to describe the large volumes of unstructured and semi-structured data that a company creates that would take too much time and money to download to a relational database for analysis. Big data analysis is often associated with cloud computing, because real-time analysis of large data sets requires frameworks like MapReduce and HBase to distribute work to tens, hundreds, or even thousands of computers. Compared with traditional data warehouse applications, big data analysis has the characteristics of large data volume and complex query and analysis. Big data requires special techniques to efficiently handle large volumes of data that tolerate elapsed time. Technologies applicable to big data, including massively parallel processing (MPP) databases, data mining grids, distributed file systems, distributed databases, cloud computing platforms, the Internet, and scalable storage systems.

ElasticSearch is a Lucene-based search server. It provides a distributed multi-user capable full-text search engine based on a RESTful web interface. Elasticsearch is developed in Java and is easy to integrate with enterprise applications. It is a popular enterprise search engine and can meet the requirements of real-time search, stability, reliability, and speed.

However, due to the underlying implementation of Elasticsearch, when the index file is too large and a large number of indexes need to be deleted, many underlying operations on the index file are required. This process takes a long time and often has a great impact on the application.

In the current IT operation and maintenance field, log analysis and monitoring tools based on the ELK (ElasticSearch+Logstash+Kibana) platform are used by more and more operation and maintenance personnel. Due to the particularity of the system and the scale of the monitored system, a large number of log files are often generated, and the timeliness requirements are high. Therefore, when the amount of data is relatively large and there are many incremental data, the index file will be large, which will affect the performance of indexing and query and put a certain pressure on the storage space. In the process of querying logs, we generally only focus on recent data, and historical data can be deleted. Therefore, how to automatically and quickly delete historical index data becomes the key to the implementation of this architecture. Based on the above situation, the present invention proposes a method for automatically cleaning and maintaining ElasticSearch log index files.

Contents of the invention

In order to make up for the defects of the prior art, the present invention provides a simple and efficient method for automatically cleaning and maintaining ElasticSearch log index files.

The present invention is achieved through the following technical solutions:

A method for automatically cleaning and maintaining ElasticSearch log index files, characterized in that: the index files are stored separately according to the time dimension, the log index deletion strategy is formulated according to business needs, and it becomes a scheduling task, and the log deletion task is scheduled using the scheduling framework , when you need to delete the historical data index, you only need to delete the index that meets the policy as a whole according to the log index deletion strategy, which can solve the efficiency problem of deleting by DeleteByquery.

The index deletion strategy formulates the log index deletion strategy according to business needs, and determines the longest valid time of the reserved index or the maximum storage space of the reserved index.

The method for automatically cleaning and maintaining the ElasticSearch log index file of the present invention comprises the following steps:

(1) Create a log index deletion policy, and create a scheduling task according to the log index deletion policy;

(2) Start the scheduling task, and execute the corresponding background task to clean up the log according to the log index deletion strategy;

(3) Determine whether the task is scheduled according to the time policy. If the task is scheduled according to the time policy, then traverse the index and delete the index that meets the time policy; if the task is not scheduled according to the time policy, delete the index according to the storage space requirements; return after deleting the index Step (2).

The beneficial effects of the present invention are: the method for automatically cleaning and maintaining the ElasticSearch log index file can quickly and efficiently delete the index file without affecting the performance of the current index and query, and solves the problem that Elasticsearch uses DeleteByquery to delete large data The problem of inefficiency in volume indexing.

Description of drawings

Accompanying drawing 1 is a schematic diagram of the method for automatically cleaning and maintaining ElasticSearch log index files according to the present invention.

detailed description

In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention clearer, the present invention will be described in detail below in conjunction with the accompanying drawings and embodiments. It should be noted that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

The method for automatically cleaning and maintaining ElasticSearch log index files stores index files separately according to the time dimension, formulates log index deletion strategies according to business needs, and makes it a scheduling task, and uses scheduling frameworks such as Quartz to schedule log deletion tasks. When you need to delete the historical data index, you only need to delete the index that meets the policy as a whole according to the log index deletion strategy, which solves the efficiency problem of deleting by DeleteByquery.

The index deletion strategy formulates the log index deletion strategy according to business needs, and determines the longest valid time of the reserved index or the maximum storage space of the reserved index.

The method for automatically cleaning and maintaining the ElasticSearch log index file of the present invention comprises the following steps:

(1) Create a log index deletion policy, and create a scheduling task according to the log index deletion policy;

(2) Start the scheduling task, and execute the corresponding background task to clean up the log according to the log index deletion strategy;

(3) Determine whether the task is scheduled according to the time policy. If the task is scheduled according to the time policy, then traverse the index and delete the index that meets the time policy; if the task is not scheduled according to the time policy, delete the index according to the storage space requirements; return after deleting the index Step (2).

Claims (3)

1. A method for automatically cleaning and maintaining ElasticSearch log index files, characterized in that the index files are stored separately according to the time dimension, the log index deletion strategy is formulated according to business needs, and it becomes a scheduling task, and the scheduling framework is used to schedule logs For deletion tasks, when you need to delete historical data indexes, you only need to delete the indexes that meet the policy as a whole according to the log index deletion strategy, which can solve the efficiency problem of deleting by DeleteByquery. 2. The method for automatically cleaning and maintaining ElasticSearch log index files according to claim 1, characterized in that: the index deletion strategy formulates the log index deletion strategy according to business needs, and determines the longest valid time for retaining the index or the duration of the retaining index. Maximum storage space. 3. according to claim 1 and the method for automatic cleaning maintenance ElasticSearch log index file, it is characterized in that, comprising the following steps: (1) Create a log index deletion policy, and create a scheduling task according to the log index deletion policy; (2) Start the scheduling task, and execute the corresponding background task to clean up the log according to the log index deletion strategy; (3) Determine whether the task is scheduled according to the time policy. If the task is scheduled according to the time policy, then traverse the index and delete the index that meets the time policy; if the task is not scheduled according to the time policy, delete the index according to the storage space requirements; return after deleting the index Step (2).