CN116340269A

CN116340269A - Method for acquiring and searching Flink task logs in real time based on elastic search

Info

Publication number: CN116340269A
Application number: CN202310321017.6A
Authority: CN
Inventors: 赵晨曦; 郭洋; 王飞; 徐旺; 陈凡
Original assignee: CLP Cloud Digital Intelligence Technology Co Ltd
Current assignee: CLP Cloud Digital Intelligence Technology Co Ltd
Priority date: 2023-03-29
Filing date: 2023-03-29
Publication date: 2023-06-27

Abstract

The invention relates to the technical field of flank tasks, and provides a method for acquiring and searching a flank task log in real time based on an elastic search, which comprises the following steps: writing a Log4j applicator component; writing a log4j.properties configuration file, and setting applicator configuration information in the written log4j.properties configuration file; submitting a Flink task, setting a user-defined JVM parameter in the task and submitting a Log4j applicator component and a log.properties configuration file synchronously; and running the Flink task, collecting the task log in real time, and carrying out real-time retrieval analysis on the collected task log. The method and the system can improve log query efficiency, help users to accurately locate key positions, display log results in a clearer mode, unify query entries and reduce query complexity.

Description

Method for acquiring and searching Flink task logs in real time based on elastic search

Technical Field

The invention relates to the technical field of flank tasks, in particular to a method for acquiring and retrieving a flank task log in real time based on an elastic search.

Background

Flink is a distributed processing engine and framework for stateful computation on bounded and unbounded data streams, which journals through the SLF4J journal interface, defaulting to Log4J2 as the underlying journal framework. JobManager and TaskManager are the primary components in the Flink, and the logs they generate are written to different text files, respectively.

In a production environment, a flank task will typically run on Yarn. Yarn is a universal resource management system that provides uniform resource management and scheduling functions for upper layer applications. Yarn provides three log viewing modes: 1. a web ui view accessing Yarn history server; 2. logging into a server where Yarn is located to execute a command: yarn logs-applicationId application _xxx view; 3. logging on to a machine running in the real state of the task, and checking the log file on the disk. In practical application, the following disadvantages exist in the above manner: 1. the log files are stored in a pure text mode and are not structured contents, retrieval is inconvenient, for example, a log containing a specified keyword and a specified log level in a certain time period needs to be queried, query data cannot be filtered when the log is checked in a web ui mode, and the check is inconvenient when the log quantity is large; when looking up in a command line mode, shell commands such as tail, grep and the like are needed, the operation is complex, and the upper threshold is high; and 2. The logs of JobManager and TaskManager can be stored in different files in a scattered manner, and a plurality of log files can be required to be checked at the same time for positioning one problem, so that the working efficiency of a user is reduced.

Therefore, how to provide an efficient and convenient method for collecting and retrieving the link task log becomes a technical problem to be solved urgently.

Disclosure of Invention

In view of the above, the invention aims to overcome the defects of the prior art and provide a method for acquiring and searching a link task log in real time based on an elastic search.

The invention provides a method for acquiring and searching a Flink task log in real time based on an elastic search, which comprises the following steps:

step S1: writing a Log4j applicator component;

step S2: writing a log4j.properties configuration file, and setting applicator configuration information in the written log4j.properties configuration file;

step S3: submitting a Flink task, setting a user-defined JVM parameter in the task and submitting a Log4j applicator component and a log.properties configuration file synchronously;

step S4: and running the Flink task, collecting the task log in real time, and carrying out real-time retrieval analysis on the collected task log.

As the preferable mode of the invention, the method for collecting and searching the Flink task log in real time based on the elastic search comprises the following steps of:

step S11: defining a log structure by writing a LogEsItem.java class file;

step S12: defining log layout by writing a FlinkLogEsLayou class file;

step S13: defining an elastic search client life cycle management object by writing an EsManager/java class file;

step S14: by writing a FlinkEsApender.java class file, log4j Appender is defined.

As the preferable mode of the invention, the method for collecting and searching the Flink task log in real time based on the elastic search comprises the following steps of: task name, task type, log content, log occurrence time stamp, and log level.

As the preferable method of the invention, the step S12 comprises the following steps:

writing a FlinkLogEsLayout file, and inheriting the attribute and the method of the Abstract StringLayout file;

setting class annotation @ Plugin on the FlinkLogEsLayout. Java class, configuring the plug-in name as FlinkLogEsLayout in the class annotation @ Plugin, and injecting the task name into the construction method of the class file through @ PlugAttribute annotation;

the method comprises the steps of rewriting a toseriarizable method in a FlinkLogEsLayou.java class file, analyzing a LogEvent object in a method parameter in the rewritten toseriarizable method, assembling the analyzed LogEvent object into a LogEsItem object, serializing the LogEsItem object into a JSON character string, and returning the JSON character string to a calling party.

As the preferable mode of the invention, the method for collecting and searching the Flink task log in real time based on the elastic search comprises the following steps of: creating an elastic search client object in a construction method of an EsManager.java class file, rewriting a releaseSub method in the EsManager.java class file, and releasing the elastic search client object in the releaseSub method.

writing a FlinkEsApender.java class file, and inheriting the attribute and the method of the Abstract appender class;

setting class annotation @ Plugin on the FlinkEsApender.java class, and configuring a plug-in name as an elastic search in the class annotation @ Plugin;

setting a Builder inner class in a FlinkEsAppender.Java class file, and setting an inner class configuration parameter in the Builder inner class, wherein the inner class configuration parameter comprises an elastic search node address, an elastic search index name, an elastic search user name and an elastic search password;

the method comprises the steps that an application method is rewritten in a FlinkEsAppend.java class file, and the send method of an EsManager class is called in the rewritten application method to write log information into an elastic search;

and (3) rewriting a stop method in the FlinkEsAppend.java class file, and calling the stop method in the EsManager class to close the elastic search client object.

As the preferable mode of the invention, the method for acquiring and searching the Flink task log in real time based on the elastic search comprises the following steps of: the app name, app type, layout type, task name, elastesearch node address, elastesearch index name, elastesearch user name, and elastesearch password.

As the preferable choice of the invention, the method for collecting and searching the Flink task log in real time based on the elastic search is characterized in that in step S3, the self-defined JVM parameter is used for transmitting the name of the Flink task to the FlinkLogEsLayout class.

As the preferable mode of the invention, the method for collecting and searching the Flink task log in real time based on the elastic search comprises the following steps of: and running a Flink task, automatically loading a Log4j applicator component, sending a task Log to an elastic search in real time according to a Log4j.properties configuration file and JVM custom parameters, and carrying out real-time retrieval analysis according to a user query request.

As the preferable mode of the invention, the method for acquiring and searching the Flink task log in real time based on the elastic search comprises the following steps of:

receiving a user query request, and dynamically constructing a REST query request of an elastic search according to query conditions in the received user query request, wherein the query conditions comprise a task name, a log starting time, a log ending time, a log level and keywords;

submitting the dynamically constructed REST query request to an elastic search for query, and displaying the query results through a browser after sequencing, wherein keywords in the query results are highlighted.

The method for acquiring and searching the Flink task log in real time based on the elastic search has the following beneficial effects:

1. the logs are stored in the elastic search in a structured mode, millisecond-level efficient full-text retrieval is carried out through the elastic search, and the log query efficiency is remarkably improved.

2. By dynamically constructing the REST query request according to the query conditions, the log query result is filtered and keyword information is highlighted, so that a user can be accurately positioned to a key position, and the log result is displayed in a clearer mode.

3. The logs of JobManager and TaskManager are collected together, so that the problem that a plurality of files need to be checked for locating one problem is avoided, the query entrance is unified, and the query complexity is reduced.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.

Fig. 1 is a schematic technical principle diagram of a method for acquiring and retrieving a link task log in real time based on an elastic search according to an exemplary first embodiment of the present invention.

Fig. 2 is a flowchart of a method for real-time collection and retrieval of a link task log based on elastic search according to an exemplary first embodiment of the present invention.

Fig. 3 is a flowchart of a method for real-time collection and retrieval of a link task log based on elastic search according to a second exemplary embodiment of the present invention.

Fig. 4 is a schematic structural diagram of the apparatus provided by the present invention.

Detailed Description

Embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

It should be noted that, without conflict, the following embodiments and features in the embodiments may be combined with each other; and, based on the embodiments in this disclosure, all other embodiments that may be made by one of ordinary skill in the art without inventive effort are within the scope of the present disclosure.

It is noted that various aspects of the embodiments are described below within the scope of the following claims. It should be apparent that the aspects described herein may be embodied in a wide variety of forms and that any specific structure and/or function described herein is merely illustrative. Based on the present disclosure, one skilled in the art will appreciate that one aspect described herein may be implemented independently of any other aspect, and that two or more of these aspects may be combined in various ways. For example, an apparatus may be implemented and/or a method practiced using any number of the aspects set forth herein. In addition, such apparatus may be implemented and/or such methods practiced using other structure and/or functionality in addition to one or more of the aspects set forth herein.

The technical principle of the invention is as follows:

the terms in the following examples are explained as follows:

elastic search: an open-source distributed, high-expansion, near-real-time search engine for fast storage, real-time retrieval and efficient analysis of massive data. The full text search can be made simple by a simple and easy-to-use restful api. The elastosearch version preferably used in the present invention is 7.8.0.

Flink task: a computing task running on the Flink platform.

Log4j applicator component: the Log4j component comprises an applicator and a Layout, wherein the applicator is responsible for outputting the Log and is used for defining the output position of the Log. The Layout is responsible for formatting the log and is used to define the output form of the log.

log4j. Properties profile: log4j is a configuration file for setting the level, the storage, and the layout of the recorder, and is set in a key=value format. By configuration, the running environment of Log4j can be created.

Log esitem. Java class file: the class definition file of the LogEsItem object.

The flinklog eslayout. Java class file: class definition file of the FlinkLogEsLayout object. The flinklog eslayout belongs to the Layout part in Log4j and is responsible for formatting the output Log.

Esmanager.java class file: the class definition file of the EaManager object. EsManager is used for lifecycle management of the elastic search client object.

The FlinkEsApender.java class file: class definition file of the FlinkEsAppender object. The FlinkEsAppender belongs to an applicator part in Log4j and is responsible for defining the position of Log output.

Log4j applicator: one of the three components of Log4j is responsible for Log output and is used for defining the Log output position.

Abstract StringLayout class: the Layout basic class provided by Log4j defines a series of commonly used attributes and methods, and the custom Layout can be conveniently realized by inheriting the attributes and the methods of the class.

Class notes @ plug in: log4j uses a plug-in mechanism to load various components and @ Plugin notation provides a convenient way to declare a class as a plug-in for Log4j. To declare information on plug-in name, plug-in type, node type, etc.

@ PluginAttribute annotation: the annotation class provided by Log4j may define parameter values that the configuration file may pass in.

toSerializable method: the basic method provided in the Abstract StringLayout class is used to convert the LogEvent object into a Log string object.

LogEvent object: log information representing individual logged events. When Log records are generated, log4j will generate this object to pass into Layout and applicator.

LogEsItem object: the objectification description of the log records facilitates the conversion and serialization of the log structures.

The releaseSub method: the basic method provided in the Abstract manager class is used to handle external resource shutdown operations when the log is shutdown.

Abstract manager class: the Manager base class provided by Log4j is used for completing management of external resources, such as Log files and elastesearch client objects. The method defines a series of commonly used attributes and methods, and can conveniently realize custom Manager by inheriting the attributes and the methods of the class.

Abstract appender class: the app foundation class provided by Log4j defines a series of commonly used attributes and methods, and the custom app can be conveniently realized by inheriting the attributes and the methods of the class.

Build internal class: the generator class provided in the Abstract apparatus class is used for analyzing the configuration in the log4j.properties configuration file to generate an apparatus object.

The application method comprises the following steps: the basic method provided in the abstract class is used to export logs to files or external systems.

The send method of the EsManager class: for sending log information to an elastic search.

Stop method in EsManager class: to close the elastesearch client object.

JVM parameters: the parameters of the Java virtual machine can be set in a format of-dkey=value, and the Java program can obtain the set parameter information at runtime.

REST query request of elastic search: a set of RESTful query requests based on the HTTP protocol can be used for realizing the access of the elastic search service by constructing the REST query requests and analyzing JSONs returned by the requests.

Fig. 1 is a technical principle schematic diagram of a method for real-time collection and retrieval of a link task log based on elastic search according to an exemplary first embodiment of the present invention, and fig. 2 is a flowchart of a method for real-time collection and retrieval of a link task log based on elastic search according to an exemplary first embodiment of the present invention, as shown in fig. 1 and 2, the method of the present embodiment is implemented in the following manner:

step S1: writing a Log4j applicator component;

The app der configuration information in step S2 of the method of this embodiment includes: the app name, app type, layout type, task name, elastesearch node address, elastesearch index name, elastesearch user name, and elastesearch password. In practical application, the configuration example of the applicator configuration information is as follows:

appender.es.name＝ElasticSearch

appender.es.type＝ElasticSearch

appender.es.layout.type＝FlinkLogEsLayout

appender.es.layout.jobName＝${sys:flink_job_name}

appender.es.nodes＝ip:port

appender.es.esIndex＝your_es_log_index

appender.es.username＝elastic

appender.es.password＝es_password

in step S3 of the method of this embodiment, the custom JVM parameter is used to pass the name of the flank task to the flanklog eslayout class.

In practical application, the custom JVM parameter is env.java.opts= -dflink_job_name=you_job_name.

According to the method, the log is stored in the elastic search in a structured mode, millisecond-level high-efficiency full-text retrieval is carried out through the elastic search, and the log query efficiency is remarkably improved. The logs of JobManager and TaskManager are collected together, so that the problem that a plurality of files need to be checked for locating one problem is avoided, the query entrance is unified, and the query complexity is reduced.

Fig. 3 is a flowchart of a method for real-time collection and retrieval of a link task log based on elastic search according to a second exemplary embodiment of the present invention, which is a preferred embodiment of the method shown in fig. 1 and 2, as shown in fig. 3, step S1 of the method of the present embodiment includes:

step S11: defining a log structure by writing a LogEsItem.java class file;

step S12: defining log layout by writing a FlinkLogEsLayou class file;

An exemplary third embodiment of the present invention provides a method for real-time collection and retrieval of a flank task log based on elastic search, which is a preferred embodiment of the method shown in fig. 1 and fig. 2.

In practical application, in the method of the present embodiment, the log structure in step S11 includes: task name, task type, log content, log occurrence time stamp, and log level.

In the method of the present embodiment, step S12 is performed as follows:

In the method of the present embodiment, step S13 is performed as follows: creating an elastic search client object in a construction method of an EsManager.java class file, rewriting a releaseSub method in the EsManager.java class file, and releasing the elastic search client object in the releaseSub method.

In the method of the present embodiment, step S14 is performed in the following manner:

An exemplary fourth embodiment of the present invention provides a method for real-time collection and retrieval of a link task log based on elastic search, which is a preferred embodiment of the method shown in fig. 1, and step S4 of the method of the present embodiment is implemented as follows: and running a Flink task, automatically loading a Log4j applicator component, sending a task Log to an elastic search in real time according to a Log4j.properties configuration file and JVM custom parameters, and carrying out real-time retrieval analysis according to a user query request.

Specifically, in the method of the embodiment, real-time search analysis is performed according to a user query request, and the method is implemented in the following manner:

According to the method, the REST query request is dynamically constructed according to the query conditions, the log query result is filtered, the keyword information is highlighted, the user can be accurately positioned to the keyword position, and the log result is displayed in a clearer mode.

As shown in fig. 4, the present invention also provides an apparatus comprising a processor 110, a communication interface 120, a memory 130 for storing a processor executable computer program, and a communication bus 140. The processor 110, the communication interface 120, and the memory 130 perform communication with each other through the communication bus 140. The processor 110 implements the above-described elastic search-based fly task log real-time acquisition and retrieval method by running an executable computer program.

Wherein the computer program in the memory 130 may be implemented in the form of software functional units and may be stored in a computer readable storage medium when sold or used as a separate product. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.

The system embodiments described above are merely illustrative, in which elements illustrated as separate elements may or may not be physically separate, and elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected based on actual needs to achieve the purpose of the embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.

From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on such understanding, the foregoing technical solutions may be embodied essentially or in part in the form of a software product, which may be stored in a computer-readable storage medium, such as a ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to perform the various embodiments or methods of some parts of the embodiments.

The foregoing is merely illustrative of the present invention, and the present invention is not limited thereto, and any changes or substitutions easily contemplated by those skilled in the art within the scope of the present invention should be included in the present invention. Therefore, the protection scope of the invention is subject to the protection scope of the claims.

Claims

1. A method for acquiring and retrieving a Flink task log in real time based on an elastic search is characterized by comprising the following steps:

step S1: writing a Log4j applicator component;

2. The method for real-time collection and retrieval of a Flink task log based on elastic search according to claim 1, wherein step S1 comprises:

step S11: defining a log structure by writing a LogEsItem.java class file;

step S12: defining log layout by writing a FlinkLogEsLayou class file;

3. The method for real-time collection and retrieval of a link task log based on elastic search according to claim 2, wherein the log structure in step S11 comprises: task name, task type, log content, log occurrence time stamp, and log level.

4. The method for real-time collection and retrieval of a link task log based on elastic search according to claim 3, wherein step S12 comprises:

5. The method for real-time collection and retrieval of a link task log based on elastic search according to claim 4, wherein step S13 comprises: creating an elastic search client object in a construction method of an EsManager.java class file, rewriting a releaseSub method in the EsManager.java class file, and releasing the elastic search client object in the releaseSub method.

6. The method for real-time collection and retrieval of a Flink task log based on elastic search according to claim 5, wherein step S14 comprises:

7. The method for real-time collection and retrieval of a link task log based on elastic search according to claim 1, wherein the applicator configuration information in step S2 comprises: the app name, app type, layout type, task name, elastesearch node address, elastesearch index name, elastesearch user name, and elastesearch password.

8. The method for real-time collection and retrieval of a flank task log based on elastic search according to claim 1, wherein in step S3, a custom JVM parameter is used to pass the name of the flank task to flanklog eslayout class.

9. The method for real-time collection and retrieval of a link task log based on elastic search according to claim 1, wherein step S4 comprises: and running a Flink task, automatically loading a Log4j applicator component, sending a task Log to an elastic search in real time according to a Log4j.properties configuration file and JVM custom parameters, and carrying out real-time retrieval analysis according to a user query request.

10. The method for real-time collection and retrieval of a link task log based on elastic search according to claim 9, wherein in step S4, real-time retrieval analysis is performed according to a user query request, comprising: