CN110727651A - Log processing method and device, terminal equipment and computer readable storage medium - Google Patents
Log processing method and device, terminal equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN110727651A CN110727651A CN201910838315.6A CN201910838315A CN110727651A CN 110727651 A CN110727651 A CN 110727651A CN 201910838315 A CN201910838315 A CN 201910838315A CN 110727651 A CN110727651 A CN 110727651A
- Authority
- CN
- China
- Prior art keywords
- grammar
- search
- syntax
- target
- specific language
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 26
- 230000006870 function Effects 0.000 claims abstract description 94
- 238000012545 processing Methods 0.000 claims abstract description 52
- 238000006243 chemical reaction Methods 0.000 claims abstract description 29
- 238000000034 method Methods 0.000 claims abstract description 19
- 230000000007 visual effect Effects 0.000 claims description 6
- 230000009466 transformation Effects 0.000 claims description 4
- 230000001131 transforming effect Effects 0.000 claims 1
- 230000008569 process Effects 0.000 abstract description 12
- 238000007405 data analysis Methods 0.000 abstract description 2
- 230000010365 information processing Effects 0.000 abstract 1
- 238000010586 diagram Methods 0.000 description 8
- 238000004458 analytical method Methods 0.000 description 7
- 238000001914 filtration Methods 0.000 description 6
- 230000002829 reductive effect Effects 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 230000003247 decreasing effect Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000000717 retained effect Effects 0.000 description 2
- 238000007619 statistical method Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 1
- 230000008094 contradictory effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/14—Details of searching files based on file metadata
- G06F16/144—Query formulation
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Library & Information Science (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention relates to the technical field of data analysis, and discloses a log processing method, which comprises the following steps: acquiring a target grammar file according to a predefined grammar format of a search processing language; carrying out syntax conversion on the target syntax file through JAVACC according to a preset syntax function to obtain a search request of a Query DSL structure; and searching according to the search request of the Query DSL structure through an Elasticissearch server to obtain a search result. The invention also provides a log processing device, terminal equipment and a computer readable storage medium. The log processing method, the terminal device and the computer readable storage medium provided by the invention can acquire the corresponding target grammar file by defining the grammar format of the search processing language, convert the target grammar file into the search request of the Query DSL structure through the grammar function, and further acquire the search result through the search request of the Query DSL structure, thereby simplifying the user operation, simplifying the operation process of log information processing and reducing the operation difficulty.
Description
Technical Field
The present invention relates to the field of data analysis technologies, and in particular, to a log processing method, a terminal device, and a computer-readable storage medium.
Background
With the continuous development of network technology, many companies carry services on the network for daily service management. In the fields of operation and maintenance and security of companies, the processing of the network access logs of the service system is particularly important, and operation and maintenance and developers can know information such as performance security of the server and access flow of websites by analyzing the logs. At present, when a business system has problems, a server can be logged in to search a log file to locate a corresponding vulnerability, and then a solution can be found. At present, when a log analysis platform is built through an open source elk (elastic search logstack kibana) to perform log information statistics, analysis and other processing, a complete Query domain specific language (Query DSL) based on JSON needs to be used to define a search statement. Due to the fact that Query DSL is very complex, search statements based on Query DSL are often very complex and have huge contents, and therefore, in the prior art, the process of inputting the search statements to the ELK log analysis platform by a user is complex in operation and consumes a lot of time.
Disclosure of Invention
In view of the above, the present invention provides a log processing method, a terminal device and a computer readable storage medium, so as to solve the problems that in the prior art, the process of inputting a search statement to an ELK log analysis platform by a user is complex in operation and consumes a lot of time.
First, in order to achieve the above object, the present invention provides a log processing method, including:
acquiring a target grammar file according to a predefined grammar format of a search processing language;
performing syntax conversion on the target syntax file through a Java compiler according to a preset syntax function to obtain a search request of a specific language structure of a query domain;
and searching according to the search request of the specific language structure of the query domain by using an Elasticissearch server to obtain a search result.
Optionally, the step of performing syntax transformation on the target syntax file through a java compiler according to a preset syntax function to obtain a search request for querying a domain specific language structure includes:
determining a target grammar function corresponding to the target grammar file from the preset grammar function through the Java compiler;
and performing syntax conversion on the target syntax file through the Java compiler according to the target syntax function to obtain a search request of the specific language structure of the query domain.
Optionally, the step of performing syntax transformation on the target syntax file according to the target syntax function by the java compiler to obtain the search request of the query domain specific language structure includes:
and compiling the target grammar file through the Java compiler to obtain target executable Java codes, and carrying out grammar conversion on the target executable Java codes according to the target grammar function to obtain the search request of the query domain specific language structure.
Optionally, the compiling the target syntax file by the java compiler to obtain a target executable java code, and performing syntax transformation on the target executable java code according to the target syntax function to obtain the search request of the query domain specific language structure includes:
under the condition that the target grammar file is a website access statistic statement, determining a website access statistic grammar function corresponding to the website access statistic statement from the preset grammar function through the Java compiler;
and carrying out grammar conversion on the website access statistics through the Java compiler according to the website access statistics grammar function to obtain the search request of the website access condition of the query domain specific language structure.
Optionally, the searching by the Elasticsearch server according to the search request of the query domain specific language structure to obtain a search result includes:
and searching by an Elasticissearch server according to the search request of the website access condition to obtain the website access condition.
Optionally, the step of obtaining the target syntax file according to a predefined syntax format of the search processing language includes:
and displaying a search box through a visual interface, and receiving a target grammar file generated according to the grammar format through the search box.
Optionally, the step of searching by the Elasticsearch server according to the search request of the query domain specific language structure to obtain the search result includes:
inputting, by the Elasticissearch search server, a search request for the query domain-specific language structure into a domain-specific language parser;
compiling the search request of the query domain specific language structure through the domain specific language parser to obtain a query domain specific language syntax tree;
and searching according to the query domain specific language syntax tree through the Elasticissearch server to obtain a search result.
In order to achieve the above object, the present invention further provides a log processing apparatus, including:
the acquisition module is used for acquiring a target grammar file according to a predefined grammar format of a search processing language;
the conversion module is used for carrying out grammar conversion on the target grammar file through a Java compiler according to a preset grammar function to obtain a search request of a specific language structure of a query domain;
and the searching module is used for searching according to the searching request of the specific language structure of the query domain through the Elasticissearch server to obtain a searching result.
In addition, in order to achieve the above object, the present invention further provides a terminal device, which includes a memory and a processor, wherein the memory stores a log processing system capable of running on the processor, and the log processing system implements the steps of the log processing method when executed by the processor.
Further, to achieve the above object, the present invention also provides a computer-readable storage medium storing a log processing system, which is executable by at least one processor to cause the at least one processor to perform the steps of the log processing method as described above.
Compared with the prior art, the log processing method, the log processing device, the terminal device and the computer readable storage medium provided by the invention can acquire the corresponding target grammar file by defining the grammar format of the search processing language, convert the target grammar file into the search request of the Query DSL structure through the grammar function, and further acquire the search result through the search request of the Query DSL structure. Therefore, a user can realize a search function by inputting a concise grammar file without providing a complex search statement, the operation process of inputting the search statement to the ELK log analysis platform by the user is simplified, and the operation time of the user is reduced.
Drawings
FIG. 1 is a diagram of an alternative hardware architecture of the terminal device of the present invention;
FIG. 2 is a schematic diagram of program modules of a first embodiment of the log processing system of the present invention;
FIG. 3 is a schematic diagram of a Query syntax format of a Query DSL structure in an embodiment of the log processing system of the present invention;
figure 4 is a schematic diagram of a DSL parser in an embodiment of the log processing system of the present invention;
FIG. 5 is a schematic diagram of a Query DSL syntax tree in an embodiment of the log processing system of the present invention;
FIG. 6 is a schematic flow chart diagram of a second embodiment of the log processing system of the present invention;
FIG. 7 is a flowchart illustrating a first embodiment of a log processing method according to the present invention;
fig. 8 is a flowchart illustrating a log processing method according to a second embodiment of the present invention.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the description relating to "first", "second", etc. in the present invention is for descriptive purposes only and is not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In addition, technical solutions between various embodiments may be combined with each other, but must be realized by a person skilled in the art, and when the technical solutions are contradictory or cannot be realized, such a combination should not be considered to exist, and is not within the protection scope of the present invention.
Fig. 1 is a schematic diagram of an alternative hardware architecture of the terminal device 2 according to the present invention.
In this embodiment, the terminal device 2 may include, but is not limited to, a memory 11, a processor 12, and a network interface 13, which may be communicatively connected to each other through a system bus. It is noted that fig. 1 only shows the terminal device 2 with components 11-13, but it is to be understood that not all shown components are required to be implemented, and that more or less components may be implemented instead.
Among other things, the terminal can be implemented in various forms. For example, the terminal described in the present invention may include mobile terminals such as a mobile phone, a tablet computer, a notebook computer, a palmtop computer, a Personal Digital Assistant (PDA), a Portable Media Player (PMP), a navigation device, a wearable device, a mobile terminal, a pedometer, and the like, and fixed terminals such as a Digital TV, a desktop computer, and the like.
While a terminal device will be exemplified in the following description, those skilled in the art will understand that the configuration according to the embodiment of the present invention can be applied to a terminal device of a fixed type in addition to elements particularly used for moving purposes.
The memory 11 includes at least one type of readable storage medium including a flash memory, a hard disk, a multimedia card, a card type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Read Only Memory (ROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a Programmable Read Only Memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the storage 11 may be an internal storage unit of the terminal device 2, such as a hard disk or a memory of the terminal device 2. In other embodiments, the memory 11 may also be an external storage device of the terminal device 2, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), or the like provided on the terminal device 2. Of course, the memory 11 may also comprise both an internal memory unit of the terminal device 2 and an external memory device thereof. In this embodiment, the memory 11 is generally used for storing an operating system installed in the terminal device 2 and various types of application software, such as a program code of the log processing system 200. Furthermore, the memory 11 may also be used to temporarily store various types of data that have been output or are to be output.
The processor 12 may be a Central Processing Unit (CPU), controller, microcontroller, microprocessor, or other data Processing chip in some embodiments. The processor 12 is typically used to control the overall operation of the terminal device 2. In this embodiment, the processor 12 is configured to run the program code stored in the memory 11 or process data, for example, run the log processing system 200.
The network interface 13 may comprise a wireless network interface or a wired network interface, and the network interface 13 is typically used for establishing communication connections between the terminal device 2 and other electronic devices.
The hardware structure and functions of the related devices of the present invention have been described in detail so far. Various embodiments of the present invention will be presented based on the above description.
First, the present invention provides a log processing system 200.
Referring to fig. 2, a program module diagram of a first embodiment of a log processing system 200 according to the present invention is shown.
In this embodiment, the log processing system 200 includes a series of computer program instructions stored on the memory 11, which when executed by the processor 12, can implement the log processing operations of the embodiments of the present invention. In some embodiments, log processing system 200 may be divided into one or more modules based on the particular operations implemented by the portions of the computer program instructions. For example, in fig. 2, the log processing system 200 may be divided into an acquisition module 201, a conversion module 202, and a search module 203. Wherein:
the obtaining module 201 is configured to obtain a target syntax file according to a predefined syntax format of a search processing language.
In this embodiment, the syntax format of the predefined search processing language includes N syntax formats. The N syntax formats include: renaming, sorting the search results, counting the search results, keeping the first M search results, and filtering the grammar format of the search results, wherein M is a positive integer greater than or equal to 1.
For example, a field of src-name may be renamed to dest-name in rename syntax format, such as when a field in a log is to be renamed. The syntax is rename < src-name > as < dest-name >, as exemplified below: .. | rename apache. For another example, for the returned search results, only the top M results need to be retained, a limit syntax format may be used, and the syntax is limit M; sorting the search results may use the sort function, with the syntax sort by [ + - ] field, + denoting increasing order, -denoting decreasing order; similarly, the syntax format of the statistical search results and the filtering search results can be set.
For another example, the syntax format defining the statistics of the IP cases accessed by the stations in one day is | stats count () byip. Stats provides the function of statistical analysis and also supports sum, min is used for solving the minimum value, max is used for solving the maximum value, avg is used for solving the average value, and grouping operation can be carried out according to the set fields.
It should be noted that the Search Processing Language (SPL) is similar to the basic syntax format of the Structured Query Language (SQL), and the Search processing Language can be defined in the syntax format based on the SQL, which is easier for operation and security personnel to use.
In this embodiment, the target grammar file is a grammar file generated according to a grammar format of a predefined search processing language, and may be, for example, "rename apache. The target grammar file is expressed by simple sentences, and the content is changed correspondingly according to the grammar format, which is not limited herein.
Optionally, the obtaining module 201 is further configured to display a search box through a visual interface, and receive, through the search box, a target syntax file generated according to the syntax format.
In this embodiment, a search box may be provided through the visual display interface, and a target grammar file input by a user according to a grammar format is received through the search box, so that user operation is facilitated.
And the conversion module 202 is configured to perform syntax conversion on the target syntax file according to a preset syntax function through a java compiler, so as to obtain a search request for querying a domain specific language structure.
In this embodiment, the preset syntax function includes N syntax function. The N grammar formats have corresponding relations with the N grammar function functions, and N is a positive integer greater than or equal to 1.
In this embodiment, the function of renaming, the function of sorting search results, the function of counting, the function of retaining the top M search results, and the function of filtering search results.
Specifically, JAVA compiler, abbreviated as JAVACC for short, is a JAVA parser generator written in JAVA language, which generates files that are pure JAVA code files, and JAVACC and its automatically generated parser can be run on multiple platforms. In this embodiment, english in the query domain-specific language is simply referred to as QueryDSL. The underlying Query DSL is JSON code, and in particular JSON (JAVA Script Object Notation) is a lightweight data exchange format. The compact and clear hierarchy makes JSON an ideal data exchange language. The method is easy to read and write, and is easy to analyze and generate by a machine, and the network transmission efficiency is effectively improved.
In this embodiment, the syntax format determines a target syntax file, and simultaneously, corresponding syntax function functions can be set in advance for different syntax files in JAVACC, and a search statement input by a user is converted into a search request of a Query DSL structure on the bottom layer of an Elasticsearch by using the corresponding syntax function functions through JAVACC. There are many grammatical functions, such as: the eval syntax function is used to add a new variable (field) to the original log, the stats syntax function is used to provide a statistics function, and the sort syntax function is used to provide a sorting function.
The searching module 203 is configured to search through the Elasticsearch server according to the search request of the query domain specific language structure, and obtain a search result.
In this embodiment, the ElasticSearch server is a Lucene-based search server that provides a distributed multi-user capable full-text search engine. Because the search request of the query DSL structure at the bottom layer of the Elasticissearch meets the format requirement of the Elasticissearch, the Elasticissearch system can determine search information such as query statements, search modes, search paths and the like according to the search request, respond to the search request and obtain related search results.
For example, suppose that a user wants to count the IP conditions of the network station access in one day and sort the IP conditions according to the request time to obtain 10 records with the maximum access times, the user only needs to input in the search box: the ElasticSearch search server carries out grammar conversion on the received input sentence to obtain a search Query grammar form shown in figure 3 so as to complete the search process.
Optionally, the searching module 203 further includes:
an input sub-module, configured to input, by the Elasticsearch server, the search request of the query domain specific language structure into a domain specific language parser;
the compiling submodule is used for compiling the search request of the query domain specific language structure through the domain specific language parser to obtain a query domain specific language syntax tree;
and the searching submodule is used for searching according to the query domain specific language syntax tree through the Elasticissearch server to obtain a searching result.
Referring to fig. 4, the domain specific language DSL parser comprises: a return result parser (SizeParser), a query parser (QueryParser), a sort parser (SortParser), and a word parser (TermParser); wherein SizeParser, is used to filter the number returned; SizeParser, which is used for sorting the returned results; term is the smallest unit of storage in the index, and Term parser is used to look up the inverted table.
Referring to fig. 5, the Query DSL syntax tree includes DSL nodes (DSLNode), Size (Size), Query (Query), Sort (Sort), Term (Term), node list (NodeList), and field nodes (FieldNode).
Therefore, the Query DSL syntax tree can be accurately obtained through the domain specific language DSL resolver, so that searching is carried out according to the Query DSL syntax tree, and a relatively accurate log searching result is obtained.
Optionally, referring to fig. 6, the conversion module 202 further includes:
a determining submodule 2021, configured to determine, by the java compiler, a target syntax function corresponding to the target syntax file from the preset syntax functions;
the conversion sub-module 2022 is configured to perform syntax conversion on the target syntax file according to the target syntax function through the java compiler, so as to obtain the search request of the query domain specific language structure.
Therefore, JAVACC can convert a simple target grammar file into a search request of a Query DSL structure through a relative target grammar function, a user does not need to provide a complicated search request of the Query DSL structure, and Query sentences input by the user are simplified, so that the user operation is simplified, and the user operation time is reduced.
Optionally, the determining sub-module 2021 is further configured to compile the target syntax file by using the java compiler to obtain a target executable java code, and perform syntax conversion on the target executable java code according to the target syntax function to obtain the search request of the query domain specific language structure.
Optionally, the determining sub-module 2021 is further configured to determine, by the java compiler, a website access statistical syntax function corresponding to the website access statistical statement from the preset syntax function, when the target syntax file is the website access statistical statement;
the conversion sub-module 2022 is further configured to perform syntax conversion on the website access statistics according to the website access statistics syntax function through the java compiler, so as to obtain a website access condition search request of the query domain specific language structure.
For example, the user wants to count the IP conditions of the website access in one day, and the user inputs in the search box: the | statscount () by ip may obtain the search request of the website access condition after the processing of the determining submodule 2021 and the converting submodule 2022.
Optionally, the searching module 203 is further configured to search through an Elasticsearch server according to the search request for the website access condition, so as to obtain the website access condition.
For example, if a website access situation search request is obtained according to "+ | stats count () by ip" input by a user, the website access situation search request is input into a domain specific language DSL parser through the Elasticsearch server, and the website access situation search request is compiled through the DSL parser to obtain the following Query DSL syntax tree;
and then searching according to the obtained Query DSL syntax tree by the Elasticissearch server to obtain a search result of the website access condition.
Therefore, the user can obtain the search result of the network access condition only by inputting a simple network access condition query statement, the search operation of the user is facilitated to be simplified, and the time of the user is saved.
The log processing device provided by the invention can acquire the corresponding target grammar file by defining the grammar format of the search processing language, convert the target grammar file into the search request of the Query DSL structure by the grammar function, and further acquire the search result by the search request of the Query DSL structure. Therefore, a user can realize a search function by inputting a concise grammar file without providing a complex search statement, the operation process of inputting the search statement to the ELK log analysis platform by the user is simplified, and the operation time of the user is reduced.
In addition, the invention also provides a log processing method.
Fig. 7 is a schematic flow chart of the log processing method according to the first embodiment of the present invention. In this embodiment, the execution order of the steps in the flowchart shown in fig. 7 may be changed and some steps may be omitted according to different requirements.
The method comprises the following steps:
step S700, according to the predefined grammar format of the search processing language, a target grammar file is obtained.
In this embodiment, the syntax format of the predefined search processing language includes N syntax formats. The N syntax formats include: renaming, sorting the search results, counting the search results, keeping the first M search results, and filtering the grammar format of the search results, wherein M is a positive integer greater than or equal to 1.
For example, a field of src-name may be renamed to dest-name in rename syntax format, such as when a field in a log is to be renamed. The syntax is rename < src-name > as < dest-name >, as exemplified below: .. | rename apache. For another example, for the returned search results, only the top M results need to be retained, a limit syntax format may be used, and the syntax is limit M; sorting the search results may use the sort function, with the syntax sort by [ + - ] field, + denoting increasing order, -denoting decreasing order; similarly, the syntax format of the statistical search results and the filtering search results can be set.
For another example, the syntax format defining the statistics of the IP cases accessed by the stations in one day is | stats count () byip. Stats provides the function of statistical analysis and also supports sum, min is used for solving the minimum value, max is used for solving the maximum value, avg is used for solving the average value, and grouping operation can be carried out according to the set fields.
It should be noted that the Search Processing Language (SPL) is similar to the basic syntax format of the Structured Query Language (SQL), and the Search processing Language can be defined in the syntax format based on the SQL, which is easier for operation and security personnel to use.
In this embodiment, the target grammar file is a grammar file generated according to a grammar format of a predefined search processing language, and may be, for example, "rename apache. The target grammar file is expressed by simple sentences, and the content is changed correspondingly according to the grammar format, which is not limited herein.
Optionally, the step S700 may include the following steps:
and displaying a search box through a visual interface, and receiving a target grammar file generated according to the grammar format through the search box.
In this embodiment, a search box may be provided through the visual display interface, and a target grammar file input by a user according to a grammar format is received through the search box, so that user operation is facilitated.
Step S702, the target grammar file is subjected to grammar conversion through a Java compiler according to a preset grammar function, and a search request of a specific language structure of a query domain is obtained.
In this embodiment, the preset syntax function includes N syntax function. The N grammar formats have corresponding relations with the N grammar function functions, and N is a positive integer greater than or equal to 1.
In this embodiment, the function of renaming, the function of sorting search results, the function of counting, the function of retaining the top M search results, and the function of filtering search results.
Specifically, JAVA compiler, abbreviated as JAVACC for short, is a JAVA parser generator written in JAVA language, which generates files that are pure JAVA code files, and JAVACC and its automatically generated parser can be run on multiple platforms. In this embodiment, english in the query domain-specific language is simply referred to as QueryDSL. The underlying Query DSL is JSON code, and in particular JSON (JAVA Script Object Notation) is a lightweight data exchange format. The compact and clear hierarchy makes JSON an ideal data exchange language. The method is easy to read and write, and is easy to analyze and generate by a machine, and the network transmission efficiency is effectively improved.
In this embodiment, the syntax format determines a target syntax file, and simultaneously, corresponding syntax function functions can be set in advance for different syntax files in JAVACC, and a search statement input by a user is converted into a search request of a Query DSL structure on the bottom layer of an Elasticsearch by using the corresponding syntax function functions through JAVACC. There are many grammatical functions, such as: the eval syntax function is used to add a new variable (field) to the original log, the stats syntax function is used to provide a statistics function, and the sort syntax function is used to provide a sorting function.
Step S704, searching is carried out through the Elasticissearch server according to the search request of the specific language structure of the query domain, and a search result is obtained.
In this embodiment, the ElasticSearch server is a Lucene-based search server that provides a distributed multi-user capable full-text search engine. Because the search request of the query DSL structure at the bottom layer of the Elasticissearch meets the format requirement of the Elasticissearch, the Elasticissearch system can determine search information such as query statements, search modes, search paths and the like according to the search request, respond to the search request and obtain related search results.
For example, suppose that a user wants to count the IP conditions of the network station access in one day and sort the IP conditions according to the request time to obtain 10 records with the maximum access times, the user only needs to input in the search box: the ElasticSearch search server carries out grammar conversion on the received input sentence to obtain a search Query grammar form shown in figure 3 so as to complete the search process.
Optionally, step S704 may include the following processes:
inputting, by the Elasticissearch search server, a search request for the query domain-specific language structure into a domain-specific language parser;
compiling the search request of the query domain specific language structure through the domain specific language parser to obtain a query domain specific language syntax tree;
and searching according to the query domain specific language syntax tree through the Elasticissearch server to obtain a search result.
Referring to fig. 4, the domain specific language DSL parser comprises: a return result parser (SizeParser), a query parser (QueryParser), a sort parser (SortParser), and a word parser (TermParser); wherein SizeParser, is used to filter the number returned; SizeParser, which is used for sorting the returned results; term is the smallest unit of storage in the index, and Term parser is used to look up the inverted table.
Referring to fig. 5, the Query DSL syntax tree includes DSL nodes (DSLNode), Size (Size), Query (Query), Sort (Sort), Term (Term), node list (NodeList), and field nodes (FieldNode).
Therefore, the Query DSL syntax tree can be accurately obtained through the domain specific language DSL resolver, so that searching is carried out according to the Query DSL syntax tree, and a relatively accurate log searching result is obtained.
Optionally, please refer to fig. 8, which is a flowchart illustrating a log processing method according to a second embodiment of the present invention. In this embodiment, step S800 of the log processing method is similar to step S700 of the first embodiment, and step S806 is similar to step S704 of the first embodiment. The difference is that steps S802-804 are the specific implementation flow of S702 of the first embodiment. Wherein:
step S802, determining a target grammar function corresponding to the target grammar file from the preset grammar function through the Java compiler;
step S804, the Java compiler performs grammar conversion on the target grammar file according to the target grammar function to obtain the search request of the specific language structure of the query domain.
Therefore, JAVACC can convert a simple target grammar file into a search request of a Query DSL structure through a relative target grammar function, a user does not need to provide a complicated search request of the Query DSL structure, and Query sentences input by the user are simplified, so that the user operation is simplified, and the user operation time is reduced.
Optionally, the step S802 may further include the following processes:
and compiling the target grammar file through the Java compiler to obtain target executable Java codes, and carrying out grammar conversion on the target executable Java codes according to the target grammar function to obtain the search request of the query domain specific language structure.
Optionally, the compiling the target syntax file by the java compiler to obtain a target executable java code, and performing syntax conversion on the target executable java code according to the target syntax function to obtain the search request of the query domain specific language structure includes:
under the condition that the target grammar file is a website access statistic statement, determining a website access statistic grammar function corresponding to the website access statistic statement from the preset grammar function through the Java compiler;
and carrying out grammar conversion on the website access statistics through the Java compiler according to the website access statistics grammar function to obtain the search request of the website access condition of the query domain specific language structure.
For example, the user wants to count the IP conditions of the website access in one day, and the user inputs in the search box: and | statscount () by ip, processing by a Java compiler to obtain a search request of the website access condition.
Optionally, the step S704 includes:
and searching by an Elasticissearch server according to the search request of the website access condition to obtain the website access condition.
For example, if a website access situation search request is obtained according to "+ | stats count () by ip" input by a user, the website access situation search request is input into a domain specific language DSL parser through the Elasticsearch server, and the website access situation search request is compiled through the DSL parser to obtain the following Query DSL syntax tree;
and then searching according to the obtained Query DSL syntax tree by the Elasticissearch server to obtain a search result of the website access condition.
Therefore, the user can obtain the search result of the network access condition only by inputting a simple network access condition query statement, the search operation of the user is facilitated to be simplified, and the time of the user is saved.
The log processing method provided by the invention can acquire the corresponding target grammar file by defining the grammar format of the search processing language, convert the target grammar file into the search request of the Query DSL structure by the grammar function, and further acquire the search result by the search request of the Query DSL structure. The user can realize the search function by inputting a concise grammar file without providing a complex search statement, thereby simplifying the operation process of inputting the search statement to the ELK log analysis platform by the user and reducing the operation time of the user.
The present invention also provides another embodiment, which is to provide a computer-readable storage medium storing a log processing program, the log processing program being executable by at least one processor to cause the at least one processor to perform the steps of the log processing method as described above.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (such as a mobile phone, a computer, a terminal device, an air conditioner, or a network device) to execute the method according to the embodiments of the present invention.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (10)
1. A method of log processing, the method comprising the steps of:
acquiring a target grammar file according to a predefined grammar format of a search processing language;
performing syntax conversion on the target syntax file through a Java compiler according to a preset syntax function to obtain a search request of a specific language structure of a query domain;
and searching according to the search request of the specific language structure of the query domain by using an Elasticissearch server to obtain a search result.
2. The log processing method as claimed in claim 1, wherein the step of converting syntax of the object syntax file according to a preset syntax function by java compiler to obtain a search request for querying a domain-specific language structure comprises:
determining a target grammar function corresponding to the target grammar file from the preset grammar function through the Java compiler;
and performing syntax conversion on the target syntax file through the Java compiler according to the target syntax function to obtain a search request of the specific language structure of the query domain.
3. A log processing method according to claim 2, wherein said step of syntactically transforming said object syntax file by said java compiler according to said object syntax function to obtain said search request for said query domain specific language structure comprises:
and compiling the target grammar file through the Java compiler to obtain target executable Java codes, and carrying out grammar conversion on the target executable Java codes according to the target grammar function to obtain the search request of the query domain specific language structure.
4. The log processing method as claimed in claim 2, wherein the compiling the target syntax file by the java compiler to obtain target executable java code, and performing syntax transformation on the target executable java code according to the target syntax function to obtain the search request of the query domain specific language structure comprises:
under the condition that the target grammar file is a website access statistic statement, determining a website access statistic grammar function corresponding to the website access statistic statement from the preset grammar function through the Java compiler;
and carrying out grammar conversion on the website access statistics through the Java compiler according to the website access statistics grammar function to obtain the search request of the website access condition of the query domain specific language structure.
5. The log processing method of claim 4, wherein the obtaining a search result by searching through an Elasticissearch server according to the search request of the query domain specific language structure comprises:
and searching by an Elasticissearch server according to the search request of the website access condition to obtain the website access condition.
6. A log processing method as claimed in any one of claims 1 to 5, wherein said step of obtaining a target grammar file according to a grammar format of a predefined search processing language comprises:
and displaying a search box through a visual interface, and receiving a target grammar file generated according to the grammar format through the search box.
7. A log processing method as claimed in any one of claims 1 to 3, wherein said step of obtaining search results by searching through an Elasticsearch server according to a search request of said query domain specific language structure comprises:
inputting, by the Elasticissearch search server, a search request for the query domain-specific language structure into a domain-specific language parser;
compiling the search request of the query domain specific language structure through the domain specific language parser to obtain a query domain specific language syntax tree;
and searching according to the query domain specific language syntax tree through the Elasticissearch server to obtain a search result.
8. A log processing apparatus, comprising:
the acquisition module is used for acquiring a target grammar file according to a predefined grammar format of a search processing language;
the conversion module is used for carrying out grammar conversion on the target grammar file through a Java compiler according to a preset grammar function to obtain a search request of a specific language structure of a query domain;
and the searching module is used for searching according to the searching request of the specific language structure of the query domain through the Elasticissearch server to obtain a searching result.
9. A terminal device, characterized in that the terminal device comprises a memory, a processor, the memory having stored thereon a log processing system executable on the processor, the log processing system when executed by the processor implementing the steps of the log processing method according to any one of claims 1-7.
10. A computer-readable storage medium storing a log processing system executable by at least one processor to cause the at least one processor to perform the steps of the log processing method as recited in any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910838315.6A CN110727651A (en) | 2019-09-05 | 2019-09-05 | Log processing method and device, terminal equipment and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910838315.6A CN110727651A (en) | 2019-09-05 | 2019-09-05 | Log processing method and device, terminal equipment and computer readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110727651A true CN110727651A (en) | 2020-01-24 |
Family
ID=69217922
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910838315.6A Pending CN110727651A (en) | 2019-09-05 | 2019-09-05 | Log processing method and device, terminal equipment and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110727651A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111240953A (en) * | 2020-03-05 | 2020-06-05 | 北京云族佳科技有限公司 | Log processing method and device and readable storage medium |
| CN112015771B (en) * | 2020-10-15 | 2021-06-29 | 北京新唐思创教育科技有限公司 | Data retrieval method and device, electronic equipment and computer storage medium |
| CN113111641A (en) * | 2021-04-20 | 2021-07-13 | 上海渠杰信息科技有限公司 | Data operation method and equipment based on full-text search engine |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1783084A (en) * | 2004-12-02 | 2006-06-07 | 微软公司 | System and method for customization of search results |
| CN108446289A (en) * | 2017-09-26 | 2018-08-24 | 北京中安智达科技有限公司 | A kind of data retrieval method for supporting heterogeneous database |
| CN109241080A (en) * | 2018-09-29 | 2019-01-18 | 焦点科技股份有限公司 | A kind of the building application method and its system of FQL query language |
-
2019
- 2019-09-05 CN CN201910838315.6A patent/CN110727651A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1783084A (en) * | 2004-12-02 | 2006-06-07 | 微软公司 | System and method for customization of search results |
| CN108446289A (en) * | 2017-09-26 | 2018-08-24 | 北京中安智达科技有限公司 | A kind of data retrieval method for supporting heterogeneous database |
| CN109241080A (en) * | 2018-09-29 | 2019-01-18 | 焦点科技股份有限公司 | A kind of the building application method and its system of FQL query language |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111240953A (en) * | 2020-03-05 | 2020-06-05 | 北京云族佳科技有限公司 | Log processing method and device and readable storage medium |
| CN112015771B (en) * | 2020-10-15 | 2021-06-29 | 北京新唐思创教育科技有限公司 | Data retrieval method and device, electronic equipment and computer storage medium |
| CN113111641A (en) * | 2021-04-20 | 2021-07-13 | 上海渠杰信息科技有限公司 | Data operation method and equipment based on full-text search engine |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109902105B (en) | Data query system, method, device and storage medium for micro-service architecture | |
| CN110727651A (en) | Log processing method and device, terminal equipment and computer readable storage medium | |
| CN109522341B (en) | Method, device and equipment for realizing SQL-based streaming data processing engine | |
| CN112015430A (en) | JavaScript code translation method and device, computer equipment and storage medium | |
| CN111241182A (en) | Data processing method and apparatus, storage medium, and electronic apparatus | |
| CN112416962A (en) | Data query method, device and storage medium | |
| CN112181924A (en) | File conversion method, device, equipment and medium | |
| CN111586695A (en) | Short message identification method and related equipment | |
| CN117093619A (en) | Rule engine processing method and device, electronic equipment and storage medium | |
| CN113934430A (en) | Data retrieval analysis method and device, electronic equipment and storage medium | |
| CN112069052A (en) | Abnormal object detection method, device, equipment and storage medium | |
| CN110209885B (en) | Graph query method and system | |
| CN114020769A (en) | Data blood margin analysis method and device and storage medium | |
| CN113609128A (en) | Method and device for generating database entity class, terminal equipment and storage medium | |
| CN114416776A (en) | Data self-defined query statistical method | |
| CN113312540A (en) | Information processing method, device, equipment, system and readable storage medium | |
| CN110471708B (en) | Method and device for acquiring configuration items based on reusable components | |
| CN113505143A (en) | Statement type conversion method and device, storage medium and electronic device | |
| CN110489163B (en) | Method, device, equipment and storage medium for identifying remote procedure call | |
| CN111580799A (en) | Domain specific language script assembling method and system | |
| CN111949254A (en) | Method, apparatus, computer device and storage medium for generating unified AST | |
| CN114780107B (en) | Grammar analysis method and device of rule running file and decision engine | |
| CN112988778A (en) | Method and device for processing database query script | |
| CN112527880B (en) | Method, device, equipment and medium for collecting metadata information of big data cluster | |
| CN112162738B (en) | Data conversion method and device, terminal equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |