CN115329049A - System slow query analysis method, device and medium - Google Patents
System slow query analysis method, device and medium Download PDFInfo
- Publication number
- CN115329049A CN115329049A CN202211128465.6A CN202211128465A CN115329049A CN 115329049 A CN115329049 A CN 115329049A CN 202211128465 A CN202211128465 A CN 202211128465A CN 115329049 A CN115329049 A CN 115329049A
- Authority
- CN
- China
- Prior art keywords
- query
- model
- overtime
- target
- task
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application relates to the field of maintenance of an ElasticSearch system, and discloses a method, a device and a medium for slow query analysis of a system, which comprise the following steps: and acquiring a target model in a model base so as to analyze the slow query task according to the target model, wherein the model base is a database for storing the overtime model corresponding to the overtime query task. And analyzing the target models according to the query conditions and the data volume of each target model and the query task samples corresponding to each target model, and acquiring model scores to eliminate overtime query tasks caused by the preemption of computing resources. And analyzing the target model with the model score higher than the threshold value to determine the time-out reason of the query task. According to the method and the device, the model of the overtime query task in the model base is analyzed and scored, so that the misjudged model in the database is removed, and the system is conveniently analyzed and improved according to the actual overtime query task.
Description
Technical Field
The application relates to the field of maintenance of an ElasticSearch system, in particular to a method, a device and a medium for slow query analysis of the system.
Background
With the development of big data, a non-relational database represented by elastic search (a distributed full-text search engine) and the like rises rapidly, the storage and query requirements of enterprises on massive document data are met, and the method is widely applied. Due to the fact that iteration of functions of products is fast, and data volume is too large, performance bottlenecks or system faults may occur in the process of writing data into the ElasticSearch system and querying data, wherein in the process of querying data, due to the fact that system nodes and data volume are large, the situation that query tasks are overtime due to slow querying easily occurs.
In specific implementation, the query task with timeout needs to be analyzed to optimize the performance of the ElasticSearch system and improve the query speed. At present, the overtime query task is mainly determined through the slow query log provided by an official party, but as each node of the ElasticSearch system independently determines the overtime query task and only records the overtime query task completed by execution, the slow query log has low timeliness and disordered storage. Meanwhile, because the resource of the ElasticSearch system is limited, when a high-load query task exists, the efficiency of other query tasks in the system is possibly reduced, so that the system judges the influenced normal query task as a slow query task, and the subsequent analysis and improvement on the ElasticSearch system are influenced.
Therefore, how to provide an accurate and efficient system slow query analysis method to prevent the misjudgment of the query task caused by insufficient system computing resources is a problem that needs to be solved by those skilled in the art.
Disclosure of Invention
The invention aims to provide a system slow query analysis method, a system slow query analysis device and a system slow query analysis medium, so that a timeout query task in an ElasticSearch system can be accurately and efficiently obtained, and the ElasticSearch system can be conveniently analyzed and improved according to the timeout query task.
In order to solve the above technical problem, the present application provides a system slow query analysis method, including:
acquiring a target model in a model library, wherein the model library is a database used for storing a timeout model corresponding to a timeout query task, the timeout query task is a query task with query time obtained by using an ElasticSearch system query interface being greater than a first threshold time, and the target model is a timeout model meeting a first preset condition in the database;
analyzing the target models according to the query conditions and the data volume of each target model and the query task samples corresponding to each target model, and acquiring model scores;
analyzing the target model with the model score higher than a threshold value to determine a timeout reason of the query task.
Preferably, the method further comprises the following steps:
acquiring the overtime query task in each ElasticSearch node in each detection period;
judging whether the overtime model corresponding to the overtime query task exists in the model library or not;
if the overtime model exists, storing the overtime query task into a sample library corresponding to the overtime model;
and if the overtime model does not exist, storing the overtime model into the model library.
Preferably, the first preset condition is that:
no analysis is performed for a second threshold time;
or the variable quantity of the corresponding query sample of the target model is larger than a sample threshold value.
Preferably, the analyzing the target models according to the query conditions and the data volume of each target model and the query task examples corresponding to each target model includes:
constructing a tree structure corresponding to the target model according to the description language of the target model to determine the query condition number and the expensive query category number of the target model;
acquiring a query task sample corresponding to the target model, and determining the total length and the data volume of a query condition according to the tree structure and the query task sample;
and calculating the model score of the target model according to the query condition number, the expensive query category number, the total query condition length and the data volume of the target model.
Preferably, the calculating the model score of the target model according to the query condition number, the expensive query category number, the total query condition length and the data volume of the target model comprises:
determining weights corresponding to the query condition number, the expensive query category number, the total query condition length and the data volume; and the weights of the expensive query category number, the data volume, the total length of the query condition and the query condition number are reduced in sequence.
Preferably, the determining whether the timeout model corresponding to the timeout inquiry task exists in the model library includes:
constructing a detection model corresponding to the overtime query task, and calculating a detection identifier of the detection model, wherein the detection identifier is MD5;
judging whether the detection identifier is the same as the identifier of each overtime model;
and if the identifier of each overtime model is different, determining that the overtime model corresponding to the overtime query task does not exist in the model library.
Preferably, after the step of analyzing the target model with the model score higher than the threshold value, the method further includes: and optimizing the query task of the ElasticSearch system according to the timeout reason.
In order to solve the above technical problem, the present application further provides a system slow query analysis device, including:
the system comprises an acquisition module, a storage module and a processing module, wherein the acquisition module is used for acquiring a target model in a model library, the model library is a database used for storing an overtime model corresponding to an overtime query task, the overtime query task is a query task which is acquired by utilizing an ElasticSearch system query interface and has the query time larger than a first threshold time, and the target model is an overtime model which meets a first preset condition in the database;
the scoring module is used for analyzing the target models according to the query conditions and the data volume of each target model and the query task samples corresponding to each target model and acquiring model scores;
and the analysis module is used for analyzing the target model with the model score higher than the threshold value so as to determine the reason of the timeout of the query task.
In order to solve the above technical problem, the present application further provides a system slow query analysis apparatus, including a memory for storing a computer program;
a processor for implementing the steps of the system slow query analysis method when executing the computer program.
In order to solve the above technical problem, the present application further provides a computer-readable storage medium, on which a computer program is stored, and the computer program, when executed by a processor, implements the steps of the system slow query analysis method.
The application provides a system slow query analysis method, which comprises the following steps: and acquiring a target model in a model base so as to analyze the slow query task according to the target model, wherein the model base is a database used for storing an overtime model corresponding to the overtime query task, the overtime query task is a query task with the query time being greater than a first threshold time, and the target model is an overtime model meeting a first preset condition in the database. And analyzing the target models according to the query conditions and the data volume of each target model and the query task samples corresponding to each target model, and obtaining model scores to eliminate overtime query tasks caused by the occupation of computing resources and prevent the query tasks from being misjudged and interfering with subsequent analysis work. And analyzing the target model with the model score higher than the threshold value to determine the time-out reason of the query task. Therefore, according to the technical scheme provided by the application, the model of the overtime query task in the model base is analyzed and scored to remove the misjudged model in the database, so that the ElasticSearch system can be analyzed and improved according to the actual overtime query task.
In addition, the application also provides a system slow query analysis device and medium, which correspond to the method and have the same effects.
Drawings
In order to more clearly illustrate the embodiments of the present application, the drawings required for the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained by those skilled in the art without inventive effort.
Fig. 1 is a flowchart of a system slow query analysis method according to an embodiment of the present disclosure;
fig. 2 is a structural diagram of a system slow query analysis device according to an embodiment of the present application;
fig. 3 is a block diagram of a system slow query analyzer according to another embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all the embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application without any creative effort belong to the protection scope of the present application.
The core of the application is to provide a system slow query analysis method, device and medium, so as to accurately and efficiently acquire a timeout query task in an ElasticSearch system, and facilitate analysis and improvement of the ElasticSearch system according to the timeout query task.
In the working process of the ElasticSearch system, the condition that the query task is overtime due to slow query easily occurs in the data query process because the related system nodes and the data volume are large. In order to optimize the ElasticSearch system, a timeout query task in the system needs to be acquired and analyzed. But the timeout query task includes: overtime tasks due to self-logic problems, and overtime tasks due to system resources being preempted. The latter does not reflect the problem in the ElasticSearch system, and therefore, the latter query task needs to be removed when analyzing the timeout query task. In order to solve the problem, an embodiment of the present application provides a system slow query analysis method, including: and acquiring a target model in a model base so as to analyze the slow query task according to the target model, wherein the model base is a database used for storing an overtime model corresponding to the overtime query task, the overtime query task is a query task with the query time being greater than a first threshold time, and the target model is an overtime model meeting a first preset condition in the database. And analyzing the target models according to the query conditions and the data volume of each target model and the query task samples corresponding to each target model, and obtaining model scores to eliminate overtime query tasks caused by the occupation of computing resources and prevent the query tasks from being misjudged and interfering with subsequent analysis work. And analyzing the target model with the model score higher than the threshold value to determine the time-out reason of the query task. Therefore, according to the technical scheme provided by the application, the model of the overtime query task in the model library is analyzed and scored to remove the misjudged model in the database, so that the ElasticSearch system can be analyzed and improved according to the actual overtime query task.
In order that those skilled in the art will better understand the disclosure, the following detailed description is given with reference to the accompanying drawings.
Fig. 1 is a flowchart of a system slow query analysis method according to an embodiment of the present application, and as shown in fig. 1, the method includes:
s10: acquiring a target model in a model library, wherein the model library is a database used for storing an overtime model corresponding to an overtime query task, the overtime query task is a query task with query time obtained by using an ElasticSearch system query interface being greater than first threshold time, and the target model is the overtime model meeting first preset conditions in the database;
in specific implementation, the slow query analysis method provided by the present application may be executed once for a model in the model library at preset time intervals, or may be executed when the variation of the model in the model library is greater than a threshold, which is not limited herein.
It is understood that the target model is a timeout model in the database, which satisfies a first preset condition, where the first preset condition includes: models which are never analyzed and scored or models which correspond to query samples and have the variation larger than the sample threshold value, and the like. It should be noted that, since the score of each target model is determined according to the model information and the query task instance, and the overall resource amount of the ElasticSearch system is also changing, the analysis and scored model needs to be analyzed again, so the first preset condition also includes the model that the analysis is not performed within the second threshold time.
It should be noted that although the ElasticSearch system provides a slow query log, due to its hysteresis, the slow query log cannot query the current incomplete timeout task, which affects the scoring result. To solve this problem, the acquisition task may be performed every preset time (e.g., 1 minute). And acquiring the query task currently being executed through a task management API of the ElasticSearch system.
The request get command is as follows: GET _ tasks & actions =:search & executed
Example of response results:
{"nodes":{"oTUltX4IQMOUUVeiohTt8A":{"name":"H5dfFeA"，"transport_address":"127.0.0.1:9300"，"host":"127.0.0.1"，"ip":127.0.0.1:9300"，"tasks":{"oTUltX4IQMOUUVeiohTt8A:464":{"node":"oTUltX4IQMOUUVeiohTt8A"，"id":464，"type":"transport"，"action":"indices:data/read/search"，"description":"indices[test]，types[test]，search_type[QUERY_THEN_FETCH]，source[{\"query\":...}]"，"start_time_in_millis":1483478610008，"running_time_in_nanos":13991383，"cancellable":true，"cancelled":false}}}}}
the description is specific query information, start _ time _ in _ millis is a query start timestamp, running _ time _ in _ nonos is query time consumption (nanoseconds), and query information with query time consumption exceeding preset time is screened.
S11: and analyzing the target models according to the query conditions and the data volume of each target model and the query task samples corresponding to each target model, and acquiring the model scores.
It can be understood that, in order to make the score more accurately reflect the quality of each target model, the query conditions and the data size of the target model and the query task examples corresponding to each target model may be combined to analyze the target model and obtain the model score. Specifically, analyzing the target model includes: constructing a tree structure corresponding to the target model according to the description language of the target model to determine the query condition number and the expensive query category number of the target model; acquiring a query task sample corresponding to the target model, and determining the total length of query conditions and the data volume according to the tree structure and the query task sample; and calculating the model score of the target model according to the number of the query conditions, the number of the expensive query categories, the total length of the query conditions and the data quantity of the target model.
S12: target models with model scores above a threshold are analyzed to determine the timeout reason for the query task.
It can be understood that after the scores of the target models are obtained, it is also necessary to determine whether the scores of the target models are greater than a threshold, and if not, it indicates that the overtime query task corresponding to the target model is overtime due to insufficient system resources, and the task does not need to be analyzed. If the time-out reason is larger than the threshold value, the fact that the time-out query task corresponding to the target model has structural or logical problems is indicated, and analysis needs to be carried out to determine the time-out reason of the query task.
It can be understood that after the timeout reason is obtained, the query task of the ElasticSearch system needs to be optimized according to the timeout reason.
The application provides a system slow query analysis method, which comprises the following steps: and acquiring a target model in a model base so as to analyze the slow query task according to the target model, wherein the model base is a database used for storing an overtime model corresponding to the overtime query task, the overtime query task is a query task with the query time larger than a first threshold time, and the target model is an overtime model meeting a first preset condition in the database. And analyzing the target models according to the query conditions and the data volume of each target model and the query task samples corresponding to each target model, and obtaining model scores to eliminate overtime query tasks caused by the occupation of computing resources and prevent the query tasks from being misjudged and interfering with subsequent analysis work. Target models with model scores above a threshold are analyzed to determine the timeout reason for the query task. Therefore, according to the technical scheme provided by the application, the model of the overtime query task in the model library is analyzed and scored to remove the misjudged model in the database, so that the ElasticSearch system can be analyzed and improved according to the actual overtime query task.
It can be understood that, in order to improve the efficiency of analyzing the query task and avoid useless work, before obtaining the model of the overtime query task, it is further necessary to determine whether an overtime model corresponding to the overtime query task exists in the model library, including: acquiring overtime query tasks in each ElasticSearch node in each detection period;
judging whether an overtime model corresponding to the overtime query task exists in the model library or not;
if the overtime model exists, storing the overtime query task into a sample library corresponding to the overtime model;
and if the overtime model does not exist, storing the overtime model into a model library.
Specifically, the step of judging whether the timeout model corresponding to the timeout query task exists in the model library includes:
constructing a detection model corresponding to the overtime query task, and calculating a detection identifier of the detection model, wherein the detection identifier is MD5;
judging whether the detection identifier is the same as the identifier of each overtime model;
and if the identifier of each overtime model is different from the identifier of each overtime model, determining that the overtime model corresponding to the overtime query task does not exist in the model library.
In specific implementation, the source content in the description is parsed, for example, the original query statement is:
{"query":{"bool":{"must":[{"term":{"name":"test"}}，{"range":{"time":{"gte":"2022-06-21 00:00:00"，"lte":"2022-06-22 00:00:00"}}}]}
}}
when the model is constructed, specific query conditions are emptied, other conditions in the non-query are removed, and the statement is as follows:
{"query":{"bool":{"must":[{"term":{"name":null}}，{"range":{"time":null}}]}}}
and judging whether the model exists in the existing model library or not, and if not, inserting the model into the model library.
As a preferred embodiment, analyzing the target models according to the query conditions and the data amount of each target model and the query task samples corresponding to each target model includes:
constructing a tree structure corresponding to the target model according to the description language of the target model to determine the query condition number and the expensive query category number of the target model;
acquiring a query task sample corresponding to the target model, and determining the total length of query conditions and the data volume according to the tree structure and the query task sample;
and calculating the model score of the target model according to the number of the query conditions, the number of the expensive query categories, the total length of the query conditions and the data amount of the target model.
In a specific implementation, analyzing the target models one by one includes analyzing 4 aspects of expensive query category number, total query condition number, query condition length and data volume, and specifically includes:
(1) Tree construction structure of model sentences
The model statement is in a JSON format, and a tree structure is constructed in the code, so that traversal and search are facilitated.
(2) Acquiring the parent field names of the leaf nodes, counting the number of expensive query types and the total query condition number, for example, the parent field names of the leaf nodes are term and range, the total query condition number is 2, and the expensive query types include: wildcard, script, fuzzy, regexp, prefix, within this range, each count is incremented by 1.
(3) And acquiring a sample list corresponding to the model, summing the conditional lengths of the leaf nodes of the samples, and calculating the average conditional length of the sample list as a whole.
The total query condition length is calculated for each sample of the same model, for example, the condition test length is 4, the gte and lte lengths in range are both 19, and the total query condition length is 23. If there are multiple samples, the average of the total length of the condition of the multiple samples is calculated.
(4) Obtaining a sample list corresponding to the model, and inquiring the data volume indexed in the sample
In the model sample, query information (descriptor) is recorded, the information includes a query index list, the data size of the index list in the ElasticSearch is queried, and the data size queried by the query sample can be known by calculating an average value.
As a preferred embodiment, the step of calculating the model score of the target model according to the query condition number, the expensive query category number, the total query condition length and the data amount of the target model comprises the following steps:
determining weights corresponding to the number of query conditions, the number of expensive query categories, the total length of the query conditions and the data volume; the weights of the expensive query category number, the data size, the total length of the query condition and the query condition number are reduced in sequence.
In a specific implementation, based on project experience, the following (adjustable) settings are set:
1 expensive query category scored 5; the highest score of the data volume is 3, 0.1 score is recorded in each 1 million of data volume, the highest score of the query condition length is 2, 0.1 score is recorded in each 128-character, the highest score of the total query condition number is 1, and 0.1 score is recorded in each 1 condition. If a model contains 1 expensive query, the data size is 3 million, the total query condition length is 1024, and the total query condition number is 5, the score value =5+ min (3, (3 million/1 million + 0.1)) + min (2, (1024/128 + 0.1)) + min (1, 5+ 0.1) =6.6.
Furthermore, after scoring is finished, the scoring results and the dimension values of the models are stored in a scoring library, so that the scoring library is sequentially ordered according to the scoring results, several overtime query tasks with the highest scoring are selected, and the overtime query tasks are combined with business key analysis.
In the foregoing embodiments, the system slow query analysis method is described in detail, and the present application also provides embodiments corresponding to the system slow query analysis device. It should be noted that the present application describes the embodiments of the apparatus portion from two perspectives, one is from the perspective of the function module, and the other is from the perspective of the hardware.
Fig. 2 is a structural diagram of a system slow query analysis device according to an embodiment of the present application, including:
the obtaining module 10 is configured to obtain a target model in a model library, where the model library is a database used for storing an overtime model corresponding to an overtime query task, the overtime query task is a query task with a query time obtained by using an ElasticSearch system query interface being greater than a first threshold time, and the target model is an overtime model in the database that meets a first preset condition.
And the scoring module 11 is configured to analyze the target models according to the query conditions and the data amount of each target model and the query task samples corresponding to each target model, and obtain model scores.
And the analysis module 12 is used for analyzing the target model with the model score higher than the threshold value so as to determine the reason for the timeout of the query task.
Since the embodiments of the apparatus portion and the method portion correspond to each other, please refer to the description of the embodiments of the method portion for the embodiments of the apparatus portion, which is not repeated here.
The application provides a system slow query analysis device, including: and acquiring a target model in a model base so as to analyze the slow query task according to the target model, wherein the model base is a database used for storing an overtime model corresponding to the overtime query task, the overtime query task is a query task with the query time being greater than a first threshold time, and the target model is an overtime model meeting a first preset condition in the database. And analyzing the target models according to the query conditions and the data volume of each target model and the query task samples corresponding to each target model, and obtaining model scores to eliminate overtime query tasks caused by the occupation of computing resources and prevent the query tasks from being misjudged and interfering with subsequent analysis work. And analyzing the target model with the model score higher than the threshold value to determine the time-out reason of the query task. Therefore, according to the technical scheme provided by the application, the model of the overtime query task in the model base is analyzed and scored to remove the misjudged model in the database, so that the ElasticSearch system can be analyzed and improved according to the actual overtime query task.
Fig. 3 is a block diagram of a system slow query analysis device according to another embodiment of the present application, and as shown in fig. 3, the system slow query analysis device includes: a memory 20 for storing a computer program;
a processor 21, configured to implement the steps of the system slow query analysis method as described in the above embodiment when executing the computer program.
The smart device provided by the embodiment may include, but is not limited to, a smart phone, a tablet computer, a notebook computer, or a desktop computer.
The processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The Processor 21 may be implemented in hardware using at least one of a Digital Signal Processor (DSP), a Field-Programmable Gate Array (FPGA), and a Programmable Logic Array (PLA). The processor 21 may also include a main processor and a coprocessor, where the main processor is a processor for Processing data in a wake state, and is also called a Central Processing Unit (CPU); a coprocessor is a low power processor for processing data in a standby state. In some embodiments, the processor 21 may be integrated with a Graphics Processing Unit (GPU) which is responsible for rendering and drawing the content required to be displayed on the display screen. In some embodiments, the processor 21 may further include an Artificial Intelligence (AI) processor for processing computing operations related to machine learning.
The memory 20 may include one or more computer-readable storage media, which may be non-transitory. Memory 20 may also include high speed random access memory, as well as non-volatile memory, such as one or more magnetic disk storage devices, flash memory storage devices. In this embodiment, the memory 20 is at least used for storing the following computer program 201, wherein after being loaded and executed by the processor 21, the computer program can implement relevant steps of the system slow query analysis method disclosed in any one of the foregoing embodiments. In addition, the resources stored in the memory 20 may also include an operating system 202, data 203, and the like, and the storage manner may be a transient storage manner or a permanent storage manner. Operating system 202 may include, among others, windows, unix, linux, and the like. Data 203 may include, but is not limited to, data provided in a system slow query analysis method, and the like.
In some embodiments, the system slow query analysis device may further include a display 22, an input/output interface 23, a communication interface 24, a power supply 25, and a communication bus 26.
Those skilled in the art will appreciate that the architecture shown in FIG. 3 does not constitute a limitation of the system slow query analysis means and may include more or fewer components than those shown.
The system slow query analysis device provided by the embodiment of the application comprises a memory and a processor, wherein when the processor executes a program stored in the memory, the following method can be realized:
and acquiring a target model in a model library, wherein the model library is a database used for storing a timeout model corresponding to a timeout query task, the timeout query task is a query task with query time obtained by using an ElasticSearch system query interface being greater than a first threshold time, and the target model is the timeout model meeting a first preset condition in the database.
And analyzing the target models according to the query conditions and the data volume of each target model and the query task samples corresponding to each target model, and acquiring the model scores.
And analyzing the target model with the model score higher than the threshold value to determine the time-out reason of the query task.
Finally, the application also provides a corresponding embodiment of the computer readable storage medium. The computer-readable storage medium has stored thereon a computer program which, when being executed by a processor, carries out the steps as set forth in the above-mentioned method embodiments.
It is to be understood that if the method in the above embodiments is implemented in the form of software functional units and sold or used as a stand-alone product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application may be embodied in the form of a software product, which is stored in a storage medium and executes all or part of the steps of the methods described in the embodiments of the present application, or all or part of the technical solutions. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
The above provides a method, an apparatus, and a medium for analyzing slow query of a system provided by the present application. The embodiments are described in a progressive mode in the specification, the emphasis of each embodiment is on the difference from the other embodiments, and the same and similar parts among the embodiments can be referred to each other. The device disclosed in the embodiment corresponds to the method disclosed in the embodiment, so that the description is simple, and the relevant points can be referred to the description of the method part. It should be noted that, for those skilled in the art, without departing from the principle of the present application, the present application can also make several improvements and modifications, and those improvements and modifications also fall into the protection scope of the claims of the present application.
It is further noted that, in the present specification, relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrases "comprising a," "8230," "8230," or "comprising" does not exclude the presence of additional like elements in a process, method, article, or apparatus that comprises the element.
Claims (10)
1. A system slow query analysis method is characterized by comprising the following steps:
acquiring a target model in a model library, wherein the model library is a database used for storing an overtime model corresponding to an overtime query task, the overtime query task is a query task with query time obtained by using an ElasticSearch system query interface being greater than first threshold time, and the target model is an overtime model meeting first preset conditions in the database;
analyzing the target models according to the query conditions and the data volume of each target model and the query task samples corresponding to each target model, and acquiring model scores;
analyzing the target model with the model score higher than a threshold value to determine a timeout reason of the query task.
2. The system slow query analysis method of claim 1, further comprising:
acquiring the overtime query task in each ElasticSearch node in each detection period;
judging whether the overtime model corresponding to the overtime query task exists in the model library or not;
if the overtime model exists, storing the overtime query task into a sample library corresponding to the overtime model;
and if the overtime model does not exist, storing the overtime model into the model library.
3. The system slow query analysis method of claim 1, wherein the first preset condition is:
no analysis is performed for a second threshold time;
or the variable quantity of the corresponding query sample of the target model is larger than a sample threshold value.
4. The system slow query analysis method of claim 1, wherein the analyzing the target models according to the query conditions and the data volume of each target model and the query task examples corresponding to each target model comprises:
constructing a tree structure corresponding to the target model according to the description language of the target model to determine the query condition number and the expensive query category number of the target model;
acquiring a query task sample corresponding to the target model, and determining the total length and the data volume of a query condition according to the tree structure and the query task sample;
calculating the model score of the target model according to the query condition number, the expensive query category number, the total query condition length and the data volume of the target model.
5. The system slow query analysis method of claim 4, wherein the calculating the model score of the target model according to the query condition number, the expensive query category number, the total query condition length, and the data volume of the target model comprises:
determining weights corresponding to the query condition number, the expensive query category number, the total query condition length and the data volume; wherein the weights of the expensive query category number, the data size, the total query condition length and the query condition number are reduced in sequence.
6. The method of claim 2, wherein the determining whether the timeout model corresponding to the timeout task exists in the model library comprises:
constructing a detection model corresponding to the overtime query task, and calculating a detection identifier of the detection model, wherein the detection identifier is MD5;
judging whether the detection identifier is the same as the identifier of each overtime model;
and if the identifier of each overtime model is different, determining that the overtime model corresponding to the overtime query task does not exist in the model library.
7. The system slow query analysis method of claim 1, wherein the step of analyzing the target model with the model score higher than a threshold value further comprises: and optimizing the query task of the ElasticSearch system according to the timeout reason.
8. A system slow query analysis device, comprising:
the system comprises an acquisition module, a storage module and a processing module, wherein the acquisition module is used for acquiring a target model in a model library, the model library is a database used for storing an overtime model corresponding to an overtime query task, the overtime query task is a query task which is acquired by utilizing an ElasticSearch system query interface and has the query time larger than a first threshold time, and the target model is an overtime model which meets a first preset condition in the database;
the scoring module is used for analyzing the target models according to the query conditions and the data volume of each target model and the query task samples corresponding to each target model and acquiring model scores;
and the analysis module is used for analyzing the target model with the model score higher than the threshold value so as to determine the reason of the timeout of the query task.
9. A system slow query analysis device comprising a memory for storing a computer program;
a processor for implementing the steps of the system slow query analysis method as claimed in any one of claims 1 to 7 when executing said computer program.
10. A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, which computer program, when being executed by a processor, carries out the steps of the system slow query analysis method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211128465.6A CN115329049A (en) | 2022-09-16 | 2022-09-16 | System slow query analysis method, device and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211128465.6A CN115329049A (en) | 2022-09-16 | 2022-09-16 | System slow query analysis method, device and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115329049A true CN115329049A (en) | 2022-11-11 |
Family
ID=83929617
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211128465.6A Pending CN115329049A (en) | 2022-09-16 | 2022-09-16 | System slow query analysis method, device and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115329049A (en) |
-
2022
- 2022-09-16 CN CN202211128465.6A patent/CN115329049A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110020422B (en) | Feature word determining method and device and server | |
| CN110019218B (en) | Data storage and query method and equipment | |
| US20070250517A1 (en) | Method and Apparatus for Autonomically Maintaining Latent Auxiliary Database Structures for Use in Executing Database Queries | |
| CN106293891B (en) | Multidimensional investment index monitoring method | |
| US20230396633A1 (en) | Method and Apparatus for Detecting Security Event, and Computer-Readable Storage Medium | |
| CN110569214A (en) | Index construction method and device for log file and electronic equipment | |
| CN108710662B (en) | Language conversion method and device, storage medium, data query system and method | |
| CN111241059B (en) | Database optimization method and device based on database | |
| CN104361115A (en) | Entry weight definition method and device based on co-clicking | |
| CN111752955A (en) | Data processing method, device, equipment and computer readable storage medium | |
| CN111460011A (en) | Page data display method and device, server and storage medium | |
| CN114490554A (en) | Data synchronization method and device, electronic equipment and storage medium | |
| CN111984625B (en) | Database load characteristic processing method and device, medium and electronic equipment | |
| US20230153286A1 (en) | Method and system for hybrid query based on cloud analysis scene, and storage medium | |
| CN110580170A (en) | software performance risk identification method and device | |
| CN115329049A (en) | System slow query analysis method, device and medium | |
| CN115599973A (en) | User crowd label screening method, system, equipment and storage medium | |
| CN114896418A (en) | Knowledge graph construction method and device, electronic equipment and storage medium | |
| CN114880308A (en) | Metadata processing method, device and medium based on big data | |
| CN115858487A (en) | Data migration method and device | |
| CN114064606A (en) | Database migration method, device, equipment, storage medium and system | |
| CN112506953A (en) | Query method, device and storage medium based on Structured Query Language (SQL) | |
| CN115422002A (en) | Gaussian database monitoring method and device | |
| CN112152873B (en) | User identification method and device, computer equipment and storage medium | |
| CN115292370A (en) | Business document data processing method, device and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |