CN113360521A - Log query method, device, equipment and storage medium - Google Patents
Log query method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN113360521A CN113360521A CN202110771839.5A CN202110771839A CN113360521A CN 113360521 A CN113360521 A CN 113360521A CN 202110771839 A CN202110771839 A CN 202110771839A CN 113360521 A CN113360521 A CN 113360521A
- Authority
- CN
- China
- Prior art keywords
- query
- sub
- query statement
- parallel
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 38
- 238000012545 processing Methods 0.000 claims abstract description 56
- 238000004458 analytical method Methods 0.000 claims description 19
- 230000002776 aggregation Effects 0.000 claims description 13
- 238000004220 aggregation Methods 0.000 claims description 13
- 230000006870 function Effects 0.000 claims description 13
- 238000006116 polymerization reaction Methods 0.000 claims description 8
- 238000004590 computer program Methods 0.000 claims description 6
- 238000000354 decomposition reaction Methods 0.000 claims description 5
- 238000007619 statistical method Methods 0.000 claims description 5
- 230000003287 optical effect Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 238000012360 testing method Methods 0.000 description 3
- 239000013307 optical fiber Substances 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 241001080526 Vertica Species 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- BPMQVOKMMQFZGV-UHFFFAOYSA-N eprazinone hydrochloride Chemical compound Cl.Cl.C=1C=CC=CC=1C(OCC)CN(CC1)CCN1CC(C)C(=O)C1=CC=CC=C1 BPMQVOKMMQFZGV-UHFFFAOYSA-N 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000013024 troubleshooting Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2462—Approximate or statistical queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2474—Sequence data queries, e.g. querying versioned data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Computational Linguistics (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Fuzzy Systems (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention discloses a log query method, a log query device, log query equipment and a storage medium. The method comprises the following steps: when a target query statement input by a user is detected, decomposing the target query statement into a plurality of sub query statements, wherein the target query statement is an SPL query statement; determining the type of each sub-query statement according to the position of each sub-query statement in the target query statement, wherein the type of each sub-query statement comprises the following steps: a parallel type and a non-parallel type; performing parallel retrieval on the log data to be processed by using parallel type sub-query statements through a parallel retrieval engine to obtain a primary retrieval result; and performing secondary processing on the primary retrieval result by using the non-parallel type sub-query statement through the central convergence engine to obtain a log query result matched with the target query statement. According to the technical scheme of the embodiment of the invention, the massive logs can be quickly inquired and analyzed according to the SPL inquiry statement without writing a complex program, so that the log inquiry result is obtained.
Description
Technical Field
The embodiment of the invention relates to the technical field of data processing, in particular to a log query method, a log query device, log query equipment and a storage medium.
Background
At present, log data can be used in aspects of troubleshooting, monitoring, safety, compliance, electronic evidence obtaining and the like, and has huge analysis value. But as the big data age comes, the number and types of logs grow rapidly, and it becomes increasingly difficult to analyze the log content and track potential problems.
In the prior art, a general log processing system in the industry adopts a Hadoop distributed scheme, and a user needs to write a complicated MapReduce program, even multiple MapReduce programs are required to process a data analysis requirement once. When log data is analyzed, because an initial analysis target is not clear, multiple trial and error are needed to analyze successfully, and a program needs to be rewritten every trial and error, the time cost and the learning cost of a user of the traditional analysis method are high.
Disclosure of Invention
Embodiments of the present invention provide a log query method, an apparatus, a device, and a storage medium, so as to implement that a massive log can be quickly queried and analyzed according to a Search Processing Language (SPL) query statement without writing a complex program, and a log query result is obtained.
In a first aspect, an embodiment of the present invention provides a log query method, including:
when a target query statement input by a user is detected, decomposing the target query statement into a plurality of sub query statements, wherein the target query statement is an SPL query statement;
determining the type of each sub-query statement according to the position of each sub-query statement in the target query statement, wherein the type of each sub-query statement comprises the following steps: a parallel type and a non-parallel type;
performing parallel retrieval on the log data to be processed matched with the target query statement by using parallel type sub-query statements through a parallel retrieval engine to obtain a primary retrieval result;
and performing secondary processing on the primary retrieval result by using the non-parallel type sub-query statement through the central convergence engine to obtain a log query result matched with the target query statement.
Optionally, when it is detected that the target query statement is input by the user, decomposing the target query statement into a plurality of sub-query statements includes:
when a target query statement input by a user is detected, identifying a pipeline symbol in the target query statement, and taking data between two adjacent pipeline symbols as a sub-query statement;
for each pipe character, the query result of the sub-query statement on the left side of the pipe character is used as the query range of the sub-query statement on the right side of the pipe character.
Optionally, determining the type of each sub-query statement according to the position of each sub-query statement in the target query statement, includes:
searching a first target sub-query statement only processed by the central convergence engine according to the sequence from left to right;
and judging the sub-query sentences before the target sub-query sentence as parallel types, and judging the sub-query sentences after the target sub-query sentence, including the target sub-query sentence, as non-parallel types.
Optionally, the parallel search engine uses parallel sub-query statements to perform parallel search on the log data to be processed matched with the target query statement, so as to obtain a preliminary search result, where the preliminary search result includes:
inputting the sub-query sentences of the parallel type into a parallel retrieval engine, and performing parallel retrieval on the log database according to each sub-query sentence through the parallel retrieval engine to obtain a query result corresponding to each sub-query sentence;
and performing statistical analysis processing on the query results of the sub query sentences through a parallel analysis engine to obtain initial retrieval results corresponding to the sub query sentences.
Optionally, the secondary processing is performed on the preliminary retrieval result by using the non-parallel type sub-query statement through the central aggregation engine, so as to obtain a log query result matched with the target query statement, where the log query result includes:
receiving, by a central aggregation engine, a preliminary retrieval result corresponding to each parallel type of first sub-query statement and a non-parallel type of second sub-query statement;
taking the initial retrieval result as a current retrieval result, and sequentially taking the second sub-query sentences as current query sentences according to the position sequence of the second sub-query sentences from left to right in the target query sentence;
executing the current query statement, performing secondary processing on the current retrieval result, updating the processing result into the current retrieval result, returning to execute the operation of taking each second sub-query statement as the current query statement in sequence according to the position sequence of each second sub-query statement from left to right in the target query statement until all the second sub-query statements are executed;
and taking the current retrieval result as a log query result matched with the target query statement.
Optionally, performing secondary processing on the current retrieval result according to the current query statement, including:
acquiring a target instruction in a current query statement according to a built-in instruction and a function of the system;
determining the instruction execution type of the current query statement according to the position relation of each target instruction in the current query statement, wherein the instruction execution type comprises the following steps: parallel flow, central flow, batch polymerization, and hierarchical polymerization;
and performing corresponding secondary processing on the current retrieval result according to the instruction execution type of the current query statement.
Optionally, the method further includes:
and if the log query result matched with the target query statement is not obtained within the preset query time, displaying the current retrieval result to the user.
In a second aspect, an embodiment of the present invention further provides a log query apparatus, including:
the decomposition module is used for decomposing the target query statement into a plurality of sub-query statements when detecting that the target query statement is input by a user, wherein the target query statement is an SPL query statement;
a type determining module, configured to determine a type of each sub-query statement according to a position of each sub-query statement in the target query statement, where the type of each sub-query statement includes: a parallel type and a non-parallel type;
the parallel retrieval module is used for performing parallel retrieval on the log data to be processed matched with the target query statement by using the parallel type sub-query statements through a parallel retrieval engine to obtain a primary retrieval result;
and the secondary processing module is used for carrying out secondary processing on the primary retrieval result by using the non-parallel type sub-query statement through the central convergence engine to obtain a log query result matched with the target query statement.
In a third aspect, an embodiment of the present invention further provides an electronic device, where the electronic device includes:
one or more processors;
a storage device for storing one or more programs,
when the one or more programs are executed by the one or more processors, the one or more processors implement the log query method provided by any embodiment of the invention.
In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the log query method provided in any embodiment of the present invention.
In the embodiment of the invention, when the SPL query statement input by a user is detected, the SPL query statement is decomposed into a plurality of sub query statements; dividing each sub query statement into a parallel type and a non-parallel type according to the position of each sub query statement in the SPL query statement; performing parallel retrieval on the log data to be processed matched with the SPL query statement by using parallel type sub-query statements through a parallel retrieval engine to obtain a primary retrieval result; through the central convergence engine, the primary retrieval result is subjected to secondary processing by using the non-parallel type sub-query statement to obtain the log query result matched with the SPL query statement, so that the problems of high time cost and high user learning cost in log analysis of mass data in the prior art are solved, a complex program is not required to be written, and the mass logs can be rapidly queried and analyzed according to the SPL query statement to obtain the log query result.
Drawings
Fig. 1 is a flowchart of a log query method in a first embodiment of the present invention;
FIG. 2 is a flowchart of a log query method according to a second embodiment of the present invention;
fig. 3 is a schematic structural diagram of a log query device in a third embodiment of the present invention;
fig. 4 is a schematic structural diagram of an electronic device in a fourth embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Example one
Fig. 1 is a flowchart of a log query method in an embodiment of the present invention, where the present embodiment is applicable to a case of performing log query on massive log data, and the method may be executed by a log query apparatus, where the apparatus may be implemented by hardware and/or software, and may generally be integrated in an electronic device providing a log query service. As shown in fig. 1, the method includes:
In this embodiment, massive log data uploaded by a user can be collected and stored. When it is detected that the user has an input operation on a query interface provided by the system, the query statement in the SPL language input by the user can be acquired as a target query statement. The SPL language is a processing language developed for searching and analyzing unstructured data such as logs, and the SPL query statement may include a nested combination of a plurality of instructions and functions.
In this embodiment, when performing log analysis query on massive log data, it may be necessary to perform multiple queries to gradually narrow the query range and find the final query result. The method has the advantages that the user operation is convenient, the user is prevented from frequently carrying out simple query operation aiming at one-time query process, and the user can be allowed to input the complex query statement comprising the nested combination of a plurality of instructions and functions, namely the target query statement at one time. After the system acquires the target query statement input by the user, the target query statement can be decomposed into a plurality of simple sub-query statements, so that the complexity of log query performed by the system is reduced, and the query speed is increased.
Optionally, when it is detected that the target query statement is input by the user, decomposing the target query statement into a plurality of sub-query statements may include: when a target query statement input by a user is detected, identifying a pipeline symbol in the target query statement, and taking data between two adjacent pipeline symbols as a sub-query statement; for each pipe character, the query result of the sub-query statement on the left side of the pipe character is used as the query range of the sub-query statement on the right side of the pipe character.
In this embodiment, the combination and nesting of the query statements may be supported by the pipe symbol "|", so when the target query statement is detected, all the pipe symbols "|" in the target query statement may be identified first, and then a portion between two adjacent pipe symbols "|" is taken as one sub-query statement, where each sub-query statement includes at least one instruction or function. Illustratively, for a query statement: the term, | makerresult count ═ 1| even hostname ═ TEST "| -dbxlookup id, time connection ═ 221_ TEST _ vertica" query ═ SELECT | -FROM TEST. The | sub-query statement 1| sub-query statement 2| sub-query statement 3. For a query statement: the logtype, apache | bucket timestamp span ═ 1h as ts | stats avg (apache. resp _ len) by hostname, ts | eval ts _ human ═ format (ts), can be resolved as: sub-query statement 1| sub-query statement 2| sub-query statement 3| sub-query statement 4.
Illustratively, for the "| sub-query statement 1| sub-query statement 2| sub-query statement 3", the query result of the sub-query statement 1 is the query range of the sub-query statement 2, that is, the system further performs query analysis on the query result of the sub-query statement 1 according to the sub-query statement 2. Therefore, except for the sub-query statements executed in parallel, the sub-query statement on the left side in the target query statement can be executed only after the sub-query statement on the right side is executed.
In this embodiment, in order to increase the log query speed, the plurality of sub query statements obtained by analysis may be classified according to whether the sub query statements can be executed in parallel, so that the system performs parallel query and grouping on log data according to the parallel type sub query statements, thereby reducing the complexity of the log query.
Optionally, determining the type of each sub-query statement according to the position of each sub-query statement in the target query statement may include: searching a first target sub-query statement only processed by the central convergence engine according to the sequence from left to right; and judging the sub-query sentences before the target sub-query sentence as parallel types, and judging the sub-query sentences after the target sub-query sentence, including the target sub-query sentence, as non-parallel types.
In this embodiment, when the sub-query statements are classified, in order to ensure that the sub-query statements on the left side in the target query statement are executed after the sub-query statements on the left side in the target query statement are executed except for the sub-query statements executed in parallel, the sub-query statements on the right side may be executed, and a first target sub-query statement that can only be processed by the central gathering engine may be searched according to a position sequence from left to right in the target query statement. At this time, all sub-query sentences before the target sub-query sentence can be executed in parallel by the parallel search engine, and therefore, the sub-query sentences before the target sub-query sentence can be determined as parallel types, and the remaining sub-query sentences of undetermined types can be determined as non-parallel types.
Because the sub-query statement includes at least one instruction, and the instruction can be divided into an instruction that can only be executed by the central gathering engine, such as an append instruction, a lookup instruction, and the like, an instruction that can only be processed by the parallel search engine, such as a query portion, and an instruction that can be processed by the central gathering engine and the parallel search engine, such as a bucket instruction, a chart instruction, an eval instruction, and the like, if an instruction that can only be executed by the central gathering engine is included in a query statement, the query statement can only be processed by the central gathering engine, and if an instruction that can only be executed by the central gathering engine is not included in the query statement, the query statement can be processed by the parallel search engine.
In the present embodiment, sub-query statements determined as non-parallel types may also include sub-query statements that can be executed in parallel. However, in order to ensure that the execution sequence of each sub-query statement is correct and avoid excessive log grouping, the present embodiment does not select the sub-query statements that can be executed in parallel from the non-parallel sub-query statements.
And step 130, performing parallel retrieval on the log data to be processed matched with the target query statement by using the parallel type sub-query statements through a parallel retrieval engine to obtain a primary retrieval result.
In this embodiment, after determining the parallel type of sub-query statements, the parallel type of sub-query statements may be input into the parallel search engine, so that the parallel search engine executes the received sub-query statements in parallel, and performs parallel query on the mass logs to be processed, thereby obtaining a plurality of log data packets.
Optionally, the parallel search engine uses parallel sub-query statements to perform parallel search on the log data to be processed matched with the target query statement, so as to obtain a preliminary search result, where the preliminary search result includes: inputting the sub-query sentences of the parallel type into a parallel retrieval engine, and performing parallel retrieval on the log database according to each sub-query sentence through the parallel retrieval engine to obtain a query result corresponding to each sub-query sentence; and performing statistical analysis processing on the query results of the sub query sentences through a parallel analysis engine to obtain initial retrieval results corresponding to the sub query sentences.
Illustratively, the sub-query statements Q1-Q3 of parallel type are input into a parallel search engine, the parallel search engine executes the sub-query statements Q1-Q3 simultaneously, and retrieves corresponding log data from a log database according to a log index, resulting in 3 sets of query results corresponding to Q1-Q3, respectively. In order to further perform statistical analysis on the query results, 3 groups of query results corresponding to Q1-Q3 may be input into a parallel analysis engine, and the query results are statistically analyzed by the parallel analysis engine, resulting in 3 log data packets corresponding to Q1-Q3.
And 140, performing secondary processing on the primary retrieval result by using the non-parallel type sub-query statement through the central convergence engine to obtain a log query result matched with the target query statement.
In this embodiment, a plurality of log data obtained by executing the sub-query statements in parallel are input into the central aggregation engine in a packet manner, the central aggregation engine sequentially executes the non-parallel sub-query statements one by one, and performs secondary data processing on each group of log data, that is, each group of log data is used as a query range, and is queried again, so as to finally obtain a log query result matched with the target query statement, and the log query result is displayed to a user.
Optionally, the performing, by the central aggregation engine, secondary processing on the preliminary search result by using the non-parallel type sub-query statement to obtain a log query result matched with the target query statement may include: receiving, by a central aggregation engine, a preliminary retrieval result corresponding to each parallel type of first sub-query statement and a non-parallel type of second sub-query statement; taking the initial retrieval result as a current retrieval result, and sequentially taking the second sub-query sentences as current query sentences according to the position sequence of the second sub-query sentences from left to right in the target query sentence; executing the current query statement, performing secondary processing on the current retrieval result, updating the processing result into the current retrieval result, returning to execute the operation of taking each second sub-query statement as the current query statement in sequence according to the position sequence of each second sub-query statement from left to right in the target query statement until all the second sub-query statements are executed; and taking the current retrieval result as a log query result matched with the target query statement.
Illustratively, by the central convergence engine, the log data groups D1-D3 and the sub-query statements Q4-Q5 of the non-parallel type are received, and then D1-D3 are taken as the current retrieval result, and Q4 is taken as the current query statement according to the position sequence from left to right in the target query statement. The central convergence engine then executes Q4 to perform logging queries again in D1-D3, respectively, resulting in new log data sets D11-D31. At this point, D11-D31 are taken as the current search result, and Q5 is updated to the current query statement. And executing the Q5 by the central convergence engine, and respectively carrying out log query again in D11-D31 to obtain new log data groups D111-D311. And D111-D311 is updated to be the current retrieval result, and because all the non-parallel type sub-query statements are completely executed at the moment, D111-D311 is the final log query result.
Optionally, performing secondary processing on the current retrieval result according to the current query statement may include: acquiring a target instruction in a current query statement according to a built-in instruction and a function of the system; determining the instruction execution type of the current query statement according to the position relation of each target instruction in the current query statement, wherein the instruction execution type comprises the following steps: parallel flow, central flow, batch polymerization, and hierarchical polymerization; and performing corresponding secondary processing on the current retrieval result according to the instruction execution type of the current query statement.
In this embodiment, in order to determine the specific meanings of the instructions and functions in the query statement and determine the instruction execution type of the query statement, the instructions and functions related to the SPL query statement are set in the system in advance. When the parallel retrieval engine or the central convergence engine queries the log according to the query statement, each instruction in the current query statement can be analyzed according to the built-in instruction and the function. And then judging the instruction execution type of the query statement according to the position relation of each instruction in the current query statement. And then according to the instruction execution type, carrying out corresponding splitting or layering and other processing on the log data, and searching the processed log data.
Illustratively, if a stats instruction in the current query statement follows a query instruction, the current query statement is of a hierarchical aggregation type; if the stats instruction in the current query statement is the following stats instruction, the current query statement is of a batch aggregation type; if the lookup instruction in the current query statement is behind the query instruction, the current query statement is of a parallel streaming type; if the lookup instruction in the current query statement is behind the stats instruction, the current query statement is of a central streaming type.
For a parallel streaming type query statement, to-be-processed log data needs to be split into different threads of different machines to be run simultaneously, and instructions are executed one by one for the log data. For the query statement of the central flow type, the log data to be processed cannot be split, but one instruction can be executed after receiving one log data. For a batch aggregation type query statement, the log data to be processed cannot be split, an instruction cannot be executed after receiving one log data, and one instruction can be executed only after all output results of the previous instruction are obtained. For a hierarchical aggregation type query statement, one log data execution instruction cannot be received, but the log data to be processed can be split into a plurality of batches, and after a first-layer intermediate result is obtained, the second layer and the third layer are converged.
Optionally, the method may further include: and if the log query result matched with the target query statement is not obtained within the preset query time, displaying the current retrieval result to the user.
In this embodiment, in order to avoid that the time for the user to wait for the query result is too long, the query time required for executing the task once is preset. If the log query task is not completed within the preset query time, namely, the log query result matched with the target query statement is not obtained, the query result is forcibly refreshed, and the current retrieval result is displayed to the user.
In addition, for the streaming processing, the user can inquire the progress of the current execution inquiry task and the temporary result of the current inquiry analysis in real time on the front-end interface, and the user can suspend the inquiry operation at any time.
In the embodiment of the invention, when the SPL query statement input by a user is detected, the SPL query statement is decomposed into a plurality of sub query statements; dividing each sub query statement into a parallel type and a non-parallel type according to the position of each sub query statement in the SPL query statement; performing parallel retrieval on the log data to be processed matched with the SPL query statement by using parallel type sub-query statements through a parallel retrieval engine to obtain a primary retrieval result; through the central convergence engine, the primary retrieval result is subjected to secondary processing by using the non-parallel type sub-query statement to obtain the log query result matched with the SPL query statement, so that the problems of high time cost and high user learning cost in log analysis of mass data in the prior art are solved, a complex program is not required to be written, and the mass logs can be rapidly queried and analyzed according to the SPL query statement to obtain the log query result.
Example two
Fig. 2 is a flowchart of a log query method in the second embodiment of the present invention, which is further detailed based on the above embodiment. The following describes a log query method provided in this embodiment with reference to fig. 2, which includes the following steps:
1. the method comprises the steps of collecting log data in real time, conducting data gathering processing such as field extraction and indexing on the log data, and storing the log data into a log database in a blocking mode according to classification information such as file names and time.
2. When detecting that a user inputs a target query statement in a query interface provided by the system, the target query statement is decomposed into a plurality of simple sub-query statements by identifying the pipeline symbols in the target query statement.
3. And searching a first target sub-query statement which can only be processed by the central convergence engine according to a position sequence from left to right in the target query statement, judging the sub-query statement before the target sub-query statement as a parallel type, and judging the remaining sub-query statements with undetermined types as non-parallel types.
4. And inputting the parallel sub-query sentences into the parallel retrieval engine and the parallel analysis engine in sequence, so that the parallel retrieval engine and the parallel analysis engine execute the received sub-query sentences in parallel, and perform parallel query on the log to obtain a plurality of log data packets.
5. And inputting a plurality of log data groups obtained by executing the sub-query sentences in parallel into a central convergence engine, executing the non-parallel sub-query sentences one by one in sequence through the central convergence engine, and inquiring again by taking each group of log data as a query range to obtain a final log query result and displaying the final log query result to a user.
If the log query task is not completed within the preset query time and the log query result matched with the target query statement is not obtained, the query result is forcibly refreshed, and the current retrieval result is displayed to the user.
In the embodiment of the invention, when the SPL query statement input by a user is detected, the SPL query statement is decomposed into a plurality of sub query statements; dividing each sub query statement into a parallel type and a non-parallel type according to the position of each sub query statement in the SPL query statement; performing parallel retrieval on the log data to be processed matched with the SPL query statement by using parallel type sub-query statements through a parallel retrieval engine to obtain a primary retrieval result; through the central convergence engine, the primary retrieval result is subjected to secondary processing by using the non-parallel type sub-query statement to obtain the log query result matched with the SPL query statement, so that the problems of high time cost and high user learning cost in log analysis of mass data in the prior art are solved, a complex program is not required to be written, and the mass logs can be rapidly queried and analyzed according to the SPL query statement to obtain the log query result.
EXAMPLE III
Fig. 3 is a schematic structural diagram of a log query device in a third embodiment of the present invention. The embodiment is applicable to the case of performing log query on massive log data, and the device can be implemented by hardware and/or software and can be generally integrated in electronic equipment providing log query service. As shown in fig. 3, the apparatus includes:
the decomposition module 310 is configured to decompose a target query statement into a plurality of sub-query statements when it is detected that a user inputs the target query statement, where the target query statement is an SPL query statement;
a type determining module 320, configured to determine a type of each sub-query statement according to a position of each sub-query statement in the target query statement, where the type of each sub-query statement includes: a parallel type and a non-parallel type;
the parallel retrieval module 330 is configured to perform parallel retrieval on the to-be-processed log data matched with the target query statement by using a parallel type sub-query statement through a parallel retrieval engine to obtain a preliminary retrieval result;
and the secondary processing module 340 is configured to perform secondary processing on the primary search result by using the non-parallel type sub-query statement through the central aggregation engine, so as to obtain a log query result matched with the target query statement.
In the embodiment of the invention, when the SPL query statement input by a user is detected, the SPL query statement is decomposed into a plurality of sub query statements; dividing each sub query statement into a parallel type and a non-parallel type according to the position of each sub query statement in the SPL query statement; performing parallel retrieval on the log data to be processed matched with the SPL query statement by using parallel type sub-query statements through a parallel retrieval engine to obtain a primary retrieval result; through the central convergence engine, the primary retrieval result is subjected to secondary processing by using the non-parallel type sub-query statement to obtain the log query result matched with the SPL query statement, so that the problems of high time cost and high user learning cost in log analysis of mass data in the prior art are solved, a complex program is not required to be written, and the mass logs can be rapidly queried and analyzed according to the SPL query statement to obtain the log query result.
Optionally, the decomposition module 310 is configured to:
when a target query statement input by a user is detected, identifying a pipeline symbol in the target query statement, and taking data between two adjacent pipeline symbols as a sub-query statement;
for each pipe character, the query result of the sub-query statement on the left side of the pipe character is used as the query range of the sub-query statement on the right side of the pipe character.
Optionally, the type determining module 320 is configured to:
searching a first target sub-query statement only processed by the central convergence engine according to the sequence from left to right;
and judging the sub-query sentences before the target sub-query sentence as parallel types, and judging the sub-query sentences after the target sub-query sentence, including the target sub-query sentence, as non-parallel types.
Optionally, the parallel retrieving module 330 is configured to:
inputting the sub-query sentences of the parallel type into a parallel retrieval engine, and performing parallel retrieval on the log database according to each sub-query sentence through the parallel retrieval engine to obtain a query result corresponding to each sub-query sentence;
and performing statistical analysis processing on the query results of the sub query sentences through a parallel analysis engine to obtain initial retrieval results corresponding to the sub query sentences.
Optionally, the secondary processing module 340 is configured to:
receiving, by a central aggregation engine, a preliminary retrieval result corresponding to each parallel type of first sub-query statement and a non-parallel type of second sub-query statement;
taking the initial retrieval result as a current retrieval result, and sequentially taking the second sub-query sentences as current query sentences according to the position sequence of the second sub-query sentences from left to right in the target query sentence;
executing the current query statement, performing secondary processing on the current retrieval result, updating the processing result into the current retrieval result, returning to execute the operation of taking each second sub-query statement as the current query statement in sequence according to the position sequence of each second sub-query statement from left to right in the target query statement until all the second sub-query statements are executed;
and taking the current retrieval result as a log query result matched with the target query statement.
Optionally, the secondary processing module 340 is specifically configured to:
acquiring a target instruction in a current query statement according to a built-in instruction and a function of the system;
determining the instruction execution type of the current query statement according to the position relation of each target instruction in the current query statement, wherein the instruction execution type comprises the following steps: parallel flow, central flow, batch polymerization, and hierarchical polymerization;
and performing corresponding secondary processing on the current retrieval result according to the instruction execution type of the current query statement.
Optionally, the method further includes:
and the forced updating module is used for displaying the current retrieval result to the user if the log query result matched with the target query statement is not obtained within the preset query time.
The log query device provided by the embodiment of the invention can execute the log query method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
Example four
Fig. 4 is a schematic structural diagram of an electronic device according to a fourth embodiment of the present invention. Fig. 4 illustrates a block diagram of an exemplary device 12 suitable for use in implementing embodiments of the present invention. The device 12 shown in fig. 4 is only an example and should not bring any limitation to the function and scope of use of the embodiments of the present invention.
As shown in FIG. 4, device 12 is in the form of a general purpose computing device. The components of device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including the system memory 28 and the processing unit 16.
The system memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM)30 and/or cache memory 32. Device 12 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 34 may be used to read from and write to non-removable, nonvolatile magnetic media (not shown in FIG. 4, and commonly referred to as a "hard drive"). Although not shown in FIG. 4, a magnetic disk drive for reading from and writing to a removable, nonvolatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, nonvolatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 by one or more data media interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the invention.
A program/utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may comprise an implementation of a network environment. Program modules 42 generally carry out the functions and/or methodologies of the described embodiments of the invention.
The processing unit 16 executes various functional applications and data processing, such as implementing the log query method provided by the embodiments of the present invention, by executing programs stored in the system memory 28.
Namely: the log query method is realized and comprises the following steps:
when a target query statement input by a user is detected, decomposing the target query statement into a plurality of sub query statements, wherein the target query statement is a Search Processing Language (SPL) query statement;
determining the type of each sub-query statement according to the position of each sub-query statement in the target query statement, wherein the type of each sub-query statement comprises the following steps: a parallel type and a non-parallel type;
performing parallel retrieval on the log data to be processed matched with the target query statement by using parallel type sub-query statements through a parallel retrieval engine to obtain a primary retrieval result;
and performing secondary processing on the primary retrieval result by using the non-parallel type sub-query statement through the central convergence engine to obtain a log query result matched with the target query statement.
EXAMPLE five
The fifth embodiment of the present invention further discloses a computer storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements a log query method, including:
when a target query statement input by a user is detected, decomposing the target query statement into a plurality of sub query statements, wherein the target query statement is a Search Processing Language (SPL) query statement;
determining the type of each sub-query statement according to the position of each sub-query statement in the target query statement, wherein the type of each sub-query statement comprises the following steps: a parallel type and a non-parallel type;
performing parallel retrieval on the log data to be processed matched with the target query statement by using parallel type sub-query statements through a parallel retrieval engine to obtain a primary retrieval result;
and performing secondary processing on the primary retrieval result by using the non-parallel type sub-query statement through the central convergence engine to obtain a log query result matched with the target query statement.
Computer storage media for embodiments of the invention may employ any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + +, or the like, as well as conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (10)
1. A log query method, comprising:
when a target query statement input by a user is detected, decomposing the target query statement into a plurality of sub query statements, wherein the target query statement is a Search Processing Language (SPL) query statement;
determining the type of each sub-query statement according to the position of each sub-query statement in the target query statement, wherein the type of each sub-query statement comprises: a parallel type and a non-parallel type;
performing parallel retrieval on the log data to be processed matched with the target query statement by using parallel type sub-query statements through a parallel retrieval engine to obtain a primary retrieval result;
and carrying out secondary processing on the primary retrieval result by using a non-parallel type sub-query statement through a central convergence engine to obtain a log query result matched with the target query statement.
2. The method of claim 1, wherein upon detecting a user input of a target query statement, decomposing the target query statement into a plurality of sub-query statements comprises:
when a target query statement input by a user is detected, identifying a pipeline symbol in the target query statement, and taking data between two adjacent pipeline symbols as a sub-query statement;
for each pipe character, the query result of the sub-query statement on the left side of the pipe character is used as the query range of the sub-query statement on the right side of the pipe character.
3. The method of claim 2, wherein determining the type of each sub-query statement based on its location in the target query statement comprises:
searching a first target sub-query statement only processed by the central convergence engine according to the sequence from left to right;
and judging the sub-query sentences before the target sub-query sentences as parallel types, and judging the sub-query sentences after the target sub-query sentences including the target sub-query sentences as non-parallel types.
4. The method of claim 1, wherein the parallel search engine uses parallel sub-query statements to perform parallel search on the log data to be processed matching the target query statement to obtain a preliminary search result, and the preliminary search result comprises:
inputting the parallel sub-query sentences into a parallel retrieval engine, and performing parallel retrieval on the log database according to the sub-query sentences through the parallel retrieval engine to obtain query results corresponding to the sub-query sentences;
and performing statistical analysis processing on the query results of the sub query sentences through a parallel analysis engine to obtain initial retrieval results corresponding to the sub query sentences.
5. The method of claim 4, wherein performing secondary processing on the preliminary search result by using a non-parallel type of sub-query statement through a central convergence engine to obtain a log query result matching the target query statement, comprises:
receiving, by a central aggregation engine, a preliminary retrieval result corresponding to each parallel type of first sub-query statement and a non-parallel type of second sub-query statement;
taking the initial retrieval result as a current retrieval result, and sequentially taking the second sub-query sentences as current query sentences according to the position sequence of the second sub-query sentences from left to right in the target query sentence;
executing the current query statement, performing secondary processing on the current retrieval result, updating the processing result into the current retrieval result, and returning to execute the operation of taking each second sub-query statement as the current query statement in sequence according to the position sequence of each second sub-query statement from left to right in the target query statement until all the second sub-query statements are executed;
and taking the current retrieval result as a log query result matched with the target query statement.
6. The method of claim 5, wherein performing secondary processing on the current search result according to the current query statement comprises:
acquiring a target instruction in a current query statement according to a built-in instruction and a function of the system;
determining the instruction execution type of the current query statement according to the position relation of each target instruction in the current query statement, wherein the instruction execution type comprises the following steps: parallel flow, central flow, batch polymerization, and hierarchical polymerization;
and performing corresponding secondary processing on the current retrieval result according to the instruction execution type of the current query statement.
7. The method of claim 5, further comprising:
and if the log query result matched with the target query statement is not obtained within the preset query time, displaying the current retrieval result to the user.
8. A log querying device, comprising:
the system comprises a decomposition module, a query module and a query module, wherein the decomposition module is used for decomposing a target query statement into a plurality of sub query statements when the target query statement input by a user is detected, and the target query statement is an SPL query statement;
a type determining module, configured to determine a type of each sub-query statement according to a position of each sub-query statement in the target query statement, where the type of each sub-query statement includes: a parallel type and a non-parallel type;
the parallel retrieval module is used for performing parallel retrieval on the log data to be processed matched with the target query statement by using the parallel type sub-query statements through a parallel retrieval engine to obtain a primary retrieval result;
and the secondary processing module is used for carrying out secondary processing on the primary retrieval result by using a non-parallel type sub-query statement through a central convergence engine to obtain a log query result matched with the target query statement.
9. An electronic device, characterized in that the device comprises:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the log query method of any of claims 1-7.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the log querying method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110771839.5A CN113360521A (en) | 2021-07-08 | 2021-07-08 | Log query method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110771839.5A CN113360521A (en) | 2021-07-08 | 2021-07-08 | Log query method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113360521A true CN113360521A (en) | 2021-09-07 |
Family
ID=77538809
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110771839.5A Pending CN113360521A (en) | 2021-07-08 | 2021-07-08 | Log query method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113360521A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114167766A (en) * | 2021-11-15 | 2022-03-11 | 成都四方伟业软件股份有限公司 | Method and device for issuing and executing instructions during multi-association equipment |
| CN114969450A (en) * | 2022-04-19 | 2022-08-30 | 北京优特捷信息技术有限公司 | User behavior analysis method, device, equipment and storage medium |
| CN116431698A (en) * | 2023-02-03 | 2023-07-14 | 北京优特捷信息技术有限公司 | Data extraction method, device, equipment and storage medium |
| CN118312533A (en) * | 2024-06-07 | 2024-07-09 | 上海数涞科技有限公司 | Query result determining method and device, electronic equipment and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106610999A (en) * | 2015-10-26 | 2017-05-03 | 北大方正集团有限公司 | Query processing method and device |
| CN108920575A (en) * | 2018-06-22 | 2018-11-30 | 北京优特捷信息技术有限公司 | Daily record data analysis method, device and readable storage medium storing program for executing based on dynamic sensing |
| US20190236185A1 (en) * | 2018-01-26 | 2019-08-01 | Vmware, Inc. | Splitting a time-range query into multiple sub-queries for serial execution |
-
2021
- 2021-07-08 CN CN202110771839.5A patent/CN113360521A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106610999A (en) * | 2015-10-26 | 2017-05-03 | 北大方正集团有限公司 | Query processing method and device |
| US20190236185A1 (en) * | 2018-01-26 | 2019-08-01 | Vmware, Inc. | Splitting a time-range query into multiple sub-queries for serial execution |
| CN108920575A (en) * | 2018-06-22 | 2018-11-30 | 北京优特捷信息技术有限公司 | Daily record data analysis method, device and readable storage medium storing program for executing based on dynamic sensing |
Non-Patent Citations (1)
| Title |
|---|
| 中山大学信息管理系: "集成化搜寻引擎", 《海峡两岸第四届图书资讯学学术研讨会论文集》 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114167766A (en) * | 2021-11-15 | 2022-03-11 | 成都四方伟业软件股份有限公司 | Method and device for issuing and executing instructions during multi-association equipment |
| CN114969450A (en) * | 2022-04-19 | 2022-08-30 | 北京优特捷信息技术有限公司 | User behavior analysis method, device, equipment and storage medium |
| CN116431698A (en) * | 2023-02-03 | 2023-07-14 | 北京优特捷信息技术有限公司 | Data extraction method, device, equipment and storage medium |
| CN116431698B (en) * | 2023-02-03 | 2024-01-30 | 北京优特捷信息技术有限公司 | Data extraction method, device, equipment and storage medium |
| CN118312533A (en) * | 2024-06-07 | 2024-07-09 | 上海数涞科技有限公司 | Query result determining method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108932294B (en) | Resume data processing method, device, equipment and storage medium based on index | |
| CN113360521A (en) | Log query method, device, equipment and storage medium | |
| CN108897867B (en) | Data processing method, device, server and medium for knowledge question answering | |
| US10657325B2 (en) | Method for parsing query based on artificial intelligence and computer device | |
| CN113760891B (en) | Data table generation method, device, equipment and storage medium | |
| US20170330084A1 (en) | Clarification of Submitted Questions in a Question and Answer System | |
| WO2021159834A1 (en) | Abnormal information processing node analysis method and apparatus, medium and electronic device | |
| CN110688544A (en) | Method, device and storage medium for querying database | |
| CN114116811B (en) | Log processing method, device, equipment and storage medium | |
| CN112445775B (en) | Fault analysis method, device, equipment and storage medium of photoetching machine | |
| CN108694221B (en) | Data real-time analysis method, module, equipment and device | |
| CN111506603B (en) | Data processing method, device, equipment and storage medium | |
| CN113792138B (en) | Report generation method and device, electronic equipment and storage medium | |
| CN111563172A (en) | Academic hotspot trend prediction method and device based on dynamic knowledge graph construction | |
| CN116150194B (en) | Data acquisition method, device, electronic equipment and computer readable medium | |
| CN111198917A (en) | Data processing method, device, equipment and storage medium | |
| CN112818070A (en) | Data query method and device based on global data dictionary and electronic equipment | |
| CN110688558B (en) | Webpage searching method, device, electronic equipment and storage medium | |
| CN110688434A (en) | Method, device, equipment and medium for processing interest points | |
| CN112487025A (en) | Data query method and device, electronic equipment and storage medium | |
| CN113138906A (en) | Call chain data acquisition method, device, equipment and storage medium | |
| CN111126034B (en) | Medical variable relation processing method and device, computer medium and electronic equipment | |
| CN114168616A (en) | Data acquisition method, device, electronic device and storage medium | |
| CN113553309A (en) | Log template determination method and device, electronic equipment and storage medium | |
| CN111400414A (en) | Decision-making method and system based on standardized enterprise data and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210907 |