CN114238395A - Database optimization method and device, electronic equipment and storage medium - Google Patents

Database optimization method and device, electronic equipment and storage medium Download PDF

Info

Publication number
CN114238395A
CN114238395A CN202210012709.8A CN202210012709A CN114238395A CN 114238395 A CN114238395 A CN 114238395A CN 202210012709 A CN202210012709 A CN 202210012709A CN 114238395 A CN114238395 A CN 114238395A
Authority
CN
China
Prior art keywords
database
processed
sql
sampling
analysis
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210012709.8A
Other languages
Chinese (zh)
Inventor
汪维
李铭
李明
唐冬平
邱峰
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Servyou Software Group Co ltd
Original Assignee
Servyou Software Group Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Servyou Software Group Co ltd filed Critical Servyou Software Group Co ltd
Priority to CN202210012709.8A priority Critical patent/CN114238395A/en
Publication of CN114238395A publication Critical patent/CN114238395A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/245Query processing
    • G06F16/2453Query optimisation
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/242Query formulation
    • G06F16/2433Query languages
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/245Query processing
    • G06F16/2455Query execution
    • G06F16/24553Query execution of query operations

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention provides a database optimization method, which comprises the following steps: acquiring SQL sentences to be processed; when detecting that the SQL sentence to be processed is executed by the database, recording execution time, and searching a corresponding entity table of the SQL sentence to be processed in the database when the execution time is determined to be greater than preset time; carrying out sampling analysis and/or histogram analysis on the entity table, and sending the obtained sampling analysis result and/or data histogram to a database so that the database carries out execution plan optimization according to the sampling analysis and/or data histogram; the method can automatically detect the condition that the database executes the SQL sentences to be processed, automatically analyze the entity table corresponding to the abnormal SQL sentences when the abnormal condition is found, does not need manual operation, can avoid the problems of low optimization efficiency and poor optimization quality caused by manual operation, and further improves the database optimization effect. The invention also provides a database optimization device, electronic equipment and a storage medium, which have the beneficial effects.

Description

Database optimization method and device, electronic equipment and storage medium
Technical Field
The present invention relates to the field of databases, and in particular, to a database optimization method and apparatus, an electronic device, and a storage medium.
Background
With the increase of the running time and the increase of the changed data, the database often runs slowly or runs slowly, and the external service capability is affected. In the related art, the problems of slow operation and unsmooth blocking of the database are usually optimized by manual intervention of operation and maintenance personnel or developers, but the method is low in efficiency, and the optimization effect is easily influenced by the optimization capabilities of the operation and maintenance personnel and the developers, so that the database optimization is difficult to meet the application requirements.
Disclosure of Invention
The invention aims to provide a database optimization method, a database optimization device, electronic equipment and a storage medium, which can automatically detect the condition that a database executes SQL sentences to be processed, automatically analyze entity tables corresponding to abnormal SQL sentences when abnormal conditions are found so as to guide the database to be optimized, and can avoid the problems of low optimization efficiency and poor optimization quality caused by manual operation.
In order to solve the above technical problem, the present invention provides a database optimization method, including:
acquiring SQL sentences to be processed;
when the SQL sentence to be processed is detected to be executed by the database, recording execution time, and searching a corresponding entity table of the SQL sentence to be processed in the database when the execution time is determined to be greater than preset time;
and carrying out sampling analysis and/or histogram analysis on the entity table, and sending the obtained sampling analysis result and/or data histogram to the database so that the database carries out plan optimization according to the sampling analysis and/or data histogram.
Optionally, the acquiring the SQL statement to be processed includes:
acquiring a preset external interface of the database, and setting an SQL statement corresponding to the preset external interface as the SQL statement to be processed;
correspondingly, when it is detected that the SQL statement to be processed is executed by the database, the recording the execution time includes:
detecting the preset external interface and recording the response time so that the external interface sends the SQL statement to be processed to the database for execution;
and when response information returned by the preset external interface is received, stopping recording the response time, and setting the response time as the execution time.
Optionally, the acquiring the SQL statement to be processed includes:
acquiring a task scheduling log generated in a preset time period, and setting SQL statements to be executed by tasks corresponding to the task scheduling log as the SQL statements to be processed; the task scheduling log records the task execution time of the SQL statement corresponding to the task executed by the database;
correspondingly, when it is detected that the SQL statement to be processed is executed by the database, the recording the execution time includes:
and setting the task execution time as the execution time.
Optionally, the setting, as the to-be-processed SQL statement, the SQL statement that the task corresponding to the task scheduling log needs to be executed includes:
searching an ETL code corresponding to the task;
and extracting the SQL sentences required to be executed by the tasks from the ETL codes, and setting the SQL sentences to be processed as the SQL sentences to be processed.
Optionally, the performing, by the database, a sampling analysis and/or a histogram analysis on the entity table, and sending an obtained sampling analysis result and/or a data histogram to the database, so that the database performs plan optimization according to the sampling analysis and/or the data histogram, includes:
carrying out first sampling analysis on the entity table according to a first preset analysis sampling degree to obtain a first sampling analysis result;
sending the first sampling analysis result to the database so that the database performs the execution plan optimization according to the first sampling analysis result;
re-executing the SQL sentence to be processed, recording first execution time, and performing second sampling analysis on the entity table according to a second preset analysis sampling degree when the first execution time is determined to be larger than the preset time to obtain a second sampling analysis result; the second preset analysis sampling degree is greater than the first preset analysis sampling degree;
sending the second sampling analysis result to the database so that the database performs the execution plan optimization according to the second sampling analysis result;
re-executing the SQL sentence to be processed and recording a second execution time, and when the second execution time is determined to be larger than the preset time, performing third sampling analysis on the entity table according to the second preset analysis sampling degree, and performing histogram analysis on the entity table to obtain a third sampling analysis result and the data histogram;
and sending the third sampling analysis result and the data histogram to the database so that the database performs the execution plan optimization according to the third sampling analysis result and the data histogram.
Optionally, after sending the third sampling analysis result and the data histogram to the database, the method further includes:
re-executing the SQL sentence to be processed, recording third execution time, and judging whether the third execution time is greater than the preset time;
if yes, outputting preset alarm information.
Optionally, after sending the third sampling analysis result and the data histogram to the database, the method further includes:
setting the SQL statement to be processed into a processed SQL statement, and generating an optimization log for the processed SQL statement;
counting the number of optimized logs corresponding to the same processed SQL statement, and judging whether the number of the optimized logs is greater than a preset threshold value or not;
if yes, the processed SQL statement is reset into the SQL statement to be processed and sent to the database for execution, and the step of recording the execution time when the SQL statement to be processed is detected to be executed by the database is carried out.
The invention also provides a database optimization device, comprising:
the acquisition module is used for acquiring the SQL sentences to be processed;
the execution time recording module is used for recording the execution time when the SQL sentence to be processed is detected to be executed by the database, and searching the entity table corresponding to the SQL sentence to be processed in the database when the execution time is determined to be greater than the preset time;
and the optimization analysis module is used for carrying out sampling analysis and/or histogram analysis on the entity table and sending the obtained sampling analysis result and/or data histogram to the database so that the database carries out plan optimization according to the sampling analysis and/or data histogram.
The present invention also provides an electronic device comprising:
a memory for storing a computer program;
a processor for implementing the database optimization method as described above when executing the computer program.
The invention also provides a storage medium, wherein the storage medium stores computer-executable instructions, and when the computer-executable instructions are loaded and executed by a processor, the database optimization method is realized.
The invention provides a database optimization method, which comprises the following steps: acquiring SQL sentences to be processed; when the SQL sentence to be processed is detected to be executed by the database, recording execution time, and searching a corresponding entity table of the SQL sentence to be processed in the database when the execution time is determined to be greater than preset time; and carrying out sampling analysis and/or histogram analysis on the entity table, and sending the obtained sampling analysis result and/or data histogram to the database so that the database carries out plan optimization according to the sampling analysis and/or data histogram.
Therefore, the method can firstly acquire the SQL sentences to be processed, which are required to be executed by the database, and record the execution time of executing the sentences by the database; then, the invention judges whether the execution time is greater than the preset time, if so, the sentence has the problem of slow execution or stuck execution in the execution process, and optimization is needed, at the moment, the invention additionally searches the entity tables corresponding to the sentence in the database, and then the entity tables perform sampling analysis or histogram analysis, wherein the sampling analysis is used for counting the data information in the entity tables, and the histogram analysis is used for determining the data distribution condition in the entity tables; after the sampling analysis result and the data histogram are obtained, the data are sent to the database, so that the database can automatically optimize the execution plan according to the sampling analysis result and the data histogram, the execution plan with the lowest execution cost is generated, and the efficiency of executing the SQL statement by the database can be improved. In other words, the method and the device can automatically detect the condition that the database executes the SQL sentences to be processed, automatically analyze the entity table corresponding to the abnormal SQL sentences when the abnormal condition is found, do not need manual operation, avoid the problems of low optimization efficiency and poor optimization quality caused by the manual operation, and further improve the optimization effect of the database. The invention also provides a database optimization device, electronic equipment and a storage medium, which have the beneficial effects.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
Fig. 1 is a flowchart of a database optimization method according to an embodiment of the present invention;
fig. 2 is a block diagram of a database optimization apparatus according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In the related art, the problems of slow operation and unsmooth blocking of the database are usually optimized by manual intervention of operation and maintenance personnel or developers, but the method is low in efficiency, and the optimization effect is easily influenced by the optimization capabilities of the operation and maintenance personnel and the developers, so that the database optimization is difficult to meet the application requirements. In view of this, the present invention provides a database optimization method, which can automatically detect a situation that a database executes an SQL statement to be processed, and automatically analyze an entity table corresponding to an abnormal SQL statement when an abnormal situation is found, so as to guide the database to perform optimization, thereby avoiding problems of low optimization efficiency and poor optimization quality caused by manual operation. Referring to fig. 1, fig. 1 is a flowchart of a database optimization method according to an embodiment of the present invention, where the method includes:
s101, obtaining the SQL statement to be processed.
SQL (Structured Query Language) is commonly used for database queries and processing. It should be noted that, the embodiment of the present invention does not limit the source of the SQL statement to be processed, and for example, the SQL statement to be processed may be from a certain interface provided by the database, or from a certain task required to be processed by the database, or may be manually collected SQL statements that need to be tested and verified, and may be set according to actual application requirements. The embodiment of the invention also does not limit the number of SQL sentences to be processed, and the number of the SQL sentences can be one or more. It can be understood that a general database interface or a database task usually executes a plurality of SQL statements in sequence, and therefore, in the embodiment of the present invention, to further achieve the effect of testing the performance of the interface or the completion quality of the database task, the SQL statements to be processed may be a plurality of SQL statements. Further, the embodiment of the present invention does not limit the specific SQL statement, and the setting may be performed according to the actual application requirements.
S102, when the SQL sentence to be processed is detected to be executed by the database, the execution time is recorded, and when the execution time is determined to be larger than the preset time, the entity table corresponding to the SQL sentence to be processed in the database is searched.
It should be noted that the embodiment of the present invention does not limit the main body of inputting the SQL statement to be processed into the database, and may be, for example, an electronic device that executes the method, or may be other devices, such as a device that executes a database task; the embodiment of the present invention also does not limit the execution subject for recording the execution time, and may be an electronic device for executing the method, or may be other devices, such as the device for executing the database task. In other words, the embodiment of the present invention does not limit the manner of determining the execution condition of the database for processing the SQL statement to be processed, and for example, the execution condition of the database may be actively determined by actively inputting the statement into the database for execution, or the execution condition of the database when receiving the statement sent by another device from another device may be passively acquired from another device. In short, it is sufficient if the execution time for the database to execute the SQL statement to be processed can be determined.
Further, when the execution time is longer than the preset time, the condition that the database runs slowly or is stuck is indicated, and the analysis and optimization of the related data tables are needed, so that when the condition that the database is determined, the corresponding entity table of the SQL statement to be processed in the database is extracted. It should be noted that the entity table is a table actually existing in the database. The embodiment of the invention does not limit the specific searching method, and can refer to the related technologies of SQL and a database.
S103, carrying out sampling analysis and/or histogram analysis on the entity table, and sending the obtained sampling analysis result and/or data histogram to the database so that the database carries out plan optimization according to the sampling analysis and/or data histogram.
After determining the corresponding entity tables, the invention performs sampling analysis, histogram analysis or simultaneous sampling analysis and histogram analysis on the tables, and returns the corresponding analysis results to the data base, so that the data base automatically optimizes according to the analysis results to determine the optimal execution plan for executing the SQL statement. The sampling analysis is used for counting data information in the entity table, the histogram analysis is used for determining the data distribution condition in the entity table, and the analysis results can help the database to better know the data distribution condition in the entity table and better optimize the data distribution condition. The embodiment of the invention does not limit the specific modes of sampling analysis and histogram analysis, and can refer to the related technology.
Further, the sampling analysis and the histogram analysis may be performed in order, considering that time and calculation resources required for the sampling analysis and the histogram analysis are different. Specifically, because sampling analysis consumes less time and occupies less resources, sampling analysis can be preferentially executed because a general database is stuck; when the sampling analysis can not obtain a better optimization effect, histogram analysis can be further added to improve the comprehensiveness and accuracy of the analysis. Further, considering that the analysis sampling rate has a large influence on the sampling analysis, when the analysis sampling rate is low, the accuracy is low but the time consumption is short, and the resource occupancy rate is low; when the analysis sampling rate is higher, although the time consumption is longer and the resource occupancy rate is higher, the analysis accuracy can be improved. Therefore, for the general slow and stuck problems, the lower analysis sampling degree can be adopted for analysis sampling; when the lower analysis sampling degree is difficult to meet the requirement, the analysis sampling degree can be increased and analysis sampling can be carried out again. It should be noted that, the embodiment of the present invention does not limit the increase amplitude and the number of increase rounds of the analysis sampling degree, and does not limit the histogram analysis to be further added when executing the analysis sampling round, and the histogram analysis can be set according to the actual application requirement. Preferably, in a possible case, two analysis sampling degrees may be preset, a first sampling analysis is performed by using the first preset analysis sampling degree, and if the analysis effect is not good, a second sampling analysis is performed by using a second larger preset analysis sampling degree; and if the analysis result still does not meet the requirement, performing third sampling analysis by using a second preset analysis sampling degree, and simultaneously adding histogram analysis. It should be noted that, the embodiment of the present invention does not limit the specific values of the first preset analysis sampling degree and the second preset analysis sampling degree, and the specific values may be set according to the actual application requirements, for example, the first preset analysis sampling degree may be 1%, and the second preset analysis sampling degree may be 10%. Further, it can be understood that the quality of the analysis result can also be measured by using the execution time of the database for executing the SQL statement to be processed, that is, after each round of analysis is completed, the SQL statement to be processed can be input into the database again for execution, and whether the corresponding execution time is greater than the preset time is determined.
In one possible case, performing sampling analysis and/or histogram analysis on the entity table, and sending the obtained sampling analysis result and/or data histogram to the database, so that the database performs plan optimization according to the sampling analysis and/or data histogram, may include:
step 11: carrying out first sampling analysis on the entity surface according to a first preset analysis sampling degree to obtain a first sampling analysis result;
step 12: sending the first sampling analysis result to a database so that the database performs execution plan optimization according to the first sampling analysis result;
step 13: re-executing the SQL sentence to be processed, recording the first execution time, and performing second sampling analysis on the entity table according to a second preset analysis sampling degree when the first execution time is determined to be larger than the preset time to obtain a second sampling analysis result; the second preset analysis sampling degree is greater than the first preset analysis sampling degree;
step 14: sending the second sampling analysis result to a database so that the database performs execution plan optimization according to the second sampling analysis result;
step 15: re-executing the SQL sentences to be processed, recording second execution time, performing third sampling analysis on the entity table according to second preset analysis sampling degree when the second execution time is determined to be larger than preset time, and performing histogram analysis on the entity table to obtain a third sampling analysis result and a data histogram;
step 16: and sending the third sampling analysis result and the data histogram to a database so that the database performs execution plan optimization according to the third sampling analysis result and the data histogram.
Of course, if the execution condition of the SQL statement to be processed still fails to reach the expected target after the third analysis is completed, it may be that the statement is abnormal, and at this time, a preset warning message may be output to remind the relevant person to perform the processing.
In one possible case, after sending the third sample analysis result and the data histogram to the database, the method may further include:
step 21: re-executing the SQL sentence to be processed, recording the third execution time, and judging whether the third execution time is greater than the preset time; if yes, go to step 22; if not, not performing operation;
step 22: and outputting preset alarm information.
Certainly, considering that the problem that the entity table corresponding to some SQL statements is slow or stuck frequently and further needs optimization processing frequently, the embodiment of the present invention may also record the number of times of optimization of the statements after the SQL statements to be processed are optimized, and actively monitor the execution condition of the statements when the number of times reaches the preset threshold value, so as to ensure the quality of the service provided by the database to the outside.
In one possible case, after sending the third sample analysis result and the data histogram to the database, the method may further include:
step 31: setting the SQL statement to be processed into a processed SQL statement, and generating an optimization log for the processed SQL statement;
step 32: counting the number of optimized logs corresponding to the same processed SQL statement, and judging whether the number of the optimized logs is greater than a preset threshold value or not; if yes, go to step 33; if not, ignoring the SQL statement to be processed;
step 33: and resetting the processed SQL sentences into SQL sentences to be processed, sending the SQL sentences to the database for execution, and recording the execution time when the SQL sentences to be processed are detected to be executed by the database.
It should be noted that, the embodiment of the present invention does not limit the specific value of the preset threshold, and the preset value may be set according to the actual application requirement.
Finally, it should be noted that the embodiment of the present invention does not limit the specific type of the database, and for example, the database may be an Oracle database or a MySQL database; in addition, the embodiment of the present invention does not limit the tool for performing the sampling analysis and the histogram analysis, for example, the tool may be an Oracle sampling tool corresponding to Oracle data, or may also be a MySQL sampling tool corresponding to MySQL, and may be selected according to the actual application requirements.
Based on the embodiment, the SQL statement to be processed, which is required to be executed by the database, is obtained, and the execution time for executing the statement by the database is recorded; then, the invention judges whether the execution time is greater than the preset time, if so, the sentence has the problem of slow execution or stuck execution in the execution process, and optimization is needed, at the moment, the invention additionally searches the entity tables corresponding to the sentence in the database, and then the entity tables perform sampling analysis or histogram analysis, wherein the sampling analysis is used for counting the data information in the entity tables, and the histogram analysis is used for determining the data distribution condition in the entity tables; after the sampling analysis result and the data histogram are obtained, the data are sent to the database, so that the database can automatically optimize the execution plan according to the sampling analysis result and the data histogram, the execution plan with the lowest execution cost is generated, and the efficiency of executing the SQL statement by the database can be improved. In other words, the method and the device can automatically detect the condition that the database executes the SQL sentences to be processed, automatically analyze the entity table corresponding to the abnormal SQL sentences when the abnormal condition is found, do not need manual operation, avoid the problems of low optimization efficiency and poor optimization quality caused by the manual operation, and further improve the optimization effect of the database.
Based on the embodiment, the invention can also detect the condition that the external interface of the database processes the SQL statement so as to ensure that the external interface can provide service to the outside with performance meeting the requirement. In one possible scenario, obtaining a to-be-processed SQL statement comprises:
s201, acquiring a preset external interface of the database, and setting an SQL statement corresponding to the preset external interface as a to-be-processed SQL statement.
It should be noted that, the embodiment of the present invention is not limited to a specific external interface, and may be set according to actual application requirements. Because the interface provides services to the outside by using a specific SQL statement, the embodiment of the invention detects the condition that the database executes the SQL statement corresponding to the interface.
Correspondingly, when it is detected that the SQL statement to be processed is executed by the database, the execution time is recorded, which includes:
s202, detecting a preset external interface and recording response time so that the external interface sends the SQL statement to be processed to a database for execution.
It should be noted that, the embodiment of the present invention does not limit the specific manner of detecting the interface, and reference may be made to the related technology of interface detection. Specifically, when the interface detects the detection, the SQL statement to be processed is sent to the database, and the response information (such as the processing result) corresponding to the database is returned to the interface using end, so that the embodiment of the present invention may stop recording the response time and set the response time as the execution time as long as the response information corresponding to the preset external interface is received.
It can be understood that, in the embodiment of the present invention, after the analysis on the entity table is completed, the preset external interface may be detected again, and the response time of the interface may be recorded again, so as to determine whether the analysis effect meets the application requirement.
And S203, stopping recording the response time when receiving the response information returned by the preset external interface, and setting the response time as the execution time.
Based on the embodiment, the invention can also detect the condition that the external interface of the database processes the SQL statement so as to ensure that the external interface can provide service to the outside with performance meeting the requirement.
Based on the embodiment, the method can also detect the condition that the database executes the database task, determine whether the SQL statement corresponding to the task has execution abnormity, and further automatically optimize in time so as to ensure that the database task can be executed with efficiency meeting the requirement. In one possible scenario, obtaining a to-be-processed SQL statement comprises:
s301, acquiring a task scheduling log generated in a preset time period, and setting SQL statements to be executed by tasks corresponding to the task scheduling log as to-be-processed SQL statements; the task scheduling log records the task execution time of the SQL statement corresponding to the database execution task.
In the embodiment of the invention, the task scheduling log is log information of scheduled execution of the database task. In order to avoid obtaining the outdated task scheduling log, the embodiment of the invention obtains the task scheduling log within the preset time period. The embodiment of the invention does not limit the specific value of the preset time period, and can be set according to the actual application requirement. Further, since the database task usually has ETL codes, where ETL represents Extract (extraction), Transform (conversion), and Load (Load), and the SQL statement corresponding to the task is usually recorded in the ETL codes, the embodiment of the present invention can Extract the SQL statement to be processed from the ETL codes corresponding to the database task in the task scheduling log.
In a possible case, setting the SQL statement that the task scheduling log corresponds to and needs to be executed by the task as the SQL statement to be processed may include:
step 41: searching an ETL code corresponding to the task;
step 42: and extracting SQL sentences required to be executed by the tasks from the ETL codes, and setting the SQL sentences to be processed.
Correspondingly, when it is detected that the SQL statement to be processed is executed by the database, the execution time is recorded, which includes:
s301, setting the task execution time as the execution time.
Because the task execution time exists in the task scheduling log, the task execution time is only required to be set as the execution time. Further, it should be noted that, because there are many SQL statements to be processed corresponding to the database task, and active re-detection will occupy a large amount of resources, in the embodiment of the present invention, the execution time corresponding to the task may be determined when a task scheduling log corresponding to the same task is obtained next time.
Based on the embodiment, the method can also detect the condition that the database executes the database task, determine whether the SQL statement corresponding to the task has execution abnormity, and further automatically optimize in time so as to ensure that the database task can be executed with efficiency meeting the requirement.
The above database optimization method is described below based on a specific example. In one possible case, the specific flow of the method is as follows:
1. starting and ending according to the set time, generally starting at 7 am and ending at 7 pm.
2. And automatically monitoring the performance of the data interface provided for the outside, and analyzing the SQL related to the interface to obtain all table names used by the SQL when finding that the interface with unsatisfactory performance exists.
3. The first analysis of these tables was done using the Oracle sampling tool, with an analysis sampling rate of 1%.
4. And after sampling is finished, testing whether the interface meets the performance requirement, if so, performing the step 9, and otherwise, performing the step 5.
5. A second analysis of these tables was performed using the Oracle sampling tool, with an analysis sampling rate of 10%.
6. And after sampling is finished, testing whether the interface meets the performance requirement, if so, performing the step 9, and otherwise, performing the step 5.
7. The tables were analyzed for a third time using an Oracle sampling tool, with an analysis sampling rate of 10%, and the data histograms were analyzed to obtain data scatter.
8. And after sampling is finished, testing whether the interface meets the performance requirement, if so, recording an optimization success log, otherwise, recording an optimization failure log, and finishing analysis.
9. The optimization of the task processing process can be carried out while the interface is optimized, a task scheduling log from yesterday night to this day is analyzed, the tasks which are slow in running and blocked are judged, related execution SQL is searched in the ETL codes according to the task names for automatic optimization, the optimization thinking is consistent with the interface optimization thinking, and the difference is that the verification result is adjusted once a day every time, and the performance test cannot be initiatively initiated.
10. In addition to passive analysis, when the analysis log reaches a certain magnitude, the tool can analyze the existing optimization condition, and actively analyze SQL which is easy to have performance problems.
In the following, the database optimization device, the electronic device, and the storage medium provided in the embodiments of the present invention are introduced, and the database optimization device, the electronic device, and the storage medium described below and the database optimization method described above may be referred to correspondingly.
Referring to fig. 2, fig. 2 is a block diagram of a database optimization apparatus according to an embodiment of the present invention, where the apparatus may include:
an obtaining module 201, configured to obtain an SQL statement to be processed;
the execution time recording module 202 is configured to record execution time when it is detected that the SQL statement to be processed is executed by the database, and search a corresponding entity table of the SQL statement to be processed in the database when it is determined that the execution time is greater than preset time;
and the optimization analysis module 203 is configured to perform sampling analysis and/or histogram analysis on the entity table, and send the obtained sampling analysis result and/or data histogram to the database, so that the database performs plan optimization according to the sampling analysis and/or data histogram.
Optionally, the obtaining module 201 may include:
the first acquisition submodule is used for acquiring a preset external interface of the database and setting an SQL statement corresponding to the preset external interface as an SQL statement to be processed;
accordingly, the execution time recording module 202 may include:
the detection submodule is used for detecting a preset external interface and recording response time so that the external interface sends the SQL statement to be processed to the database for execution;
and the timing submodule is used for stopping recording the response time when receiving the response information returned by the preset external interface and setting the response time as the execution time.
Optionally, the obtaining module 201 may include:
the second obtaining submodule is used for obtaining the task scheduling log generated in a preset time period and setting SQL sentences to be executed by the tasks corresponding to the task scheduling log as SQL sentences to be processed; the task scheduling log records the task execution time of SQL statements corresponding to the database execution task;
accordingly, the execution time recording module 202 may include:
and the setting submodule is used for setting the task execution time as the execution time.
Optionally, the second obtaining sub-module includes:
the searching unit is used for searching an ETL code corresponding to the task;
and the extraction unit is used for extracting the SQL sentences required to be executed by the tasks in the ETL codes and setting the SQL sentences to be processed.
Optionally, the optimization analysis module 203 comprises:
the first analysis submodule is used for carrying out first sampling analysis on the entity surface according to a first preset analysis sampling degree to obtain a first sampling analysis result;
the first optimization submodule is used for sending the first sampling analysis result to the database so that the database performs execution plan optimization according to the first sampling analysis result;
the second analysis submodule is used for re-executing the SQL sentence to be processed, recording the first execution time and carrying out second sampling analysis on the entity table according to a second preset analysis sampling degree when the first execution time is determined to be larger than the preset time so as to obtain a second sampling analysis result; the second preset analysis sampling degree is greater than the first preset analysis sampling degree;
the second optimization submodule is used for sending the second sampling analysis result to the database so that the database performs execution plan optimization according to the second sampling analysis result;
the second analysis submodule is used for re-executing the SQL sentence to be processed, recording second execution time, performing third sampling analysis on the entity table according to the second preset analysis sampling degree when the second execution time is determined to be larger than the preset time, and performing histogram analysis on the entity table to obtain a third sampling analysis result and a data histogram;
and the third optimization sub-module is used for sending the third sampling analysis result and the data histogram to the database so that the database performs execution plan optimization according to the third sampling analysis result and the data histogram.
Optionally, the optimization analysis module 203 may further include:
the judging submodule is used for re-executing the SQL sentence to be processed, recording the third execution time and judging whether the third execution time is greater than the preset time;
and the alarm submodule is used for outputting preset alarm information if the preset alarm information is true.
Optionally, the optimization analysis module 203 may further include:
the optimized log generation submodule is used for setting the SQL statement to be processed into a processed SQL statement and generating an optimized log for the processed SQL statement;
the statistic submodule is used for counting the number of optimized logs corresponding to the same processed SQL statement and judging whether the number of the optimized logs is larger than a preset threshold value or not;
and the setting submodule is used for resetting the processed SQL statement into the SQL statement to be processed and sending the SQL statement to the database for execution if the SQL statement to be processed is detected to be executed by the database, and recording the execution time when the SQL statement to be processed is detected to be executed by the database.
An embodiment of the present invention further provides an electronic device, including:
a memory for storing a computer program;
a processor for implementing the steps of the database optimization method as described above when executing the computer program.
Since the embodiment of the electronic device portion corresponds to the embodiment of the database optimization method portion, please refer to the description of the embodiment of the database optimization method portion for the embodiment of the electronic device portion, which is not repeated here.
The embodiment of the present invention further provides a storage medium, where a computer program is stored on the storage medium, and when the computer program is executed by a processor, the steps of the database optimization method according to any of the above embodiments are implemented.
Since the embodiment of the storage medium portion corresponds to the embodiment of the database optimization method portion, please refer to the description of the embodiment of the database optimization method portion for the embodiment of the storage medium portion, which is not repeated here.
The embodiments are described in a progressive manner in the specification, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
The above detailed description is provided for a database optimization method, an apparatus, an electronic device and a storage medium provided by the present invention. The principles and embodiments of the present invention are explained herein using specific examples, which are presented only to assist in understanding the method and its core concepts. It should be noted that, for those skilled in the art, it is possible to make various improvements and modifications to the present invention without departing from the principle of the present invention, and those improvements and modifications also fall within the scope of the claims of the present invention.

Claims (10)

1. A method of database optimization, comprising:
acquiring SQL sentences to be processed;
when the SQL sentence to be processed is detected to be executed by the database, recording execution time, and searching a corresponding entity table of the SQL sentence to be processed in the database when the execution time is determined to be greater than preset time;
and carrying out sampling analysis and/or histogram analysis on the entity table, and sending the obtained sampling analysis result and/or data histogram to the database so that the database carries out plan optimization according to the sampling analysis and/or data histogram.
2. The database optimization method according to claim 1, wherein the obtaining the SQL statement to be processed comprises:
acquiring a preset external interface of the database, and setting an SQL statement corresponding to the preset external interface as the SQL statement to be processed;
correspondingly, when it is detected that the SQL statement to be processed is executed by the database, the recording the execution time includes:
detecting the preset external interface and recording the response time so that the external interface sends the SQL statement to be processed to the database for execution;
and when response information returned by the preset external interface is received, stopping recording the response time, and setting the response time as the execution time.
3. The database optimization method according to claim 1, wherein the obtaining the SQL statement to be processed comprises:
acquiring a task scheduling log generated in a preset time period, and setting SQL statements to be executed by tasks corresponding to the task scheduling log as the SQL statements to be processed; the task scheduling log records the task execution time of the SQL statement corresponding to the task executed by the database;
correspondingly, when it is detected that the SQL statement to be processed is executed by the database, the recording the execution time includes:
and setting the task execution time as the execution time.
4. The database optimization method according to claim 3, wherein the setting of the SQL statement that the task corresponding to the task scheduling log needs to be executed as the SQL statement to be processed includes:
searching an ETL code corresponding to the task;
and extracting the SQL sentences required to be executed by the tasks from the ETL codes, and setting the SQL sentences to be processed as the SQL sentences to be processed.
5. The database optimization method according to any one of claims 1 to 4, wherein the performing sampling analysis and/or histogram analysis on the entity table and sending the obtained sampling analysis result and/or data histogram to the database so that the database performs plan optimization according to the sampling analysis and/or data histogram includes:
carrying out first sampling analysis on the entity table according to a first preset analysis sampling degree to obtain a first sampling analysis result;
sending the first sampling analysis result to the database so that the database performs the execution plan optimization according to the first sampling analysis result;
re-executing the SQL sentence to be processed, recording first execution time, and performing second sampling analysis on the entity table according to a second preset analysis sampling degree when the first execution time is determined to be larger than the preset time to obtain a second sampling analysis result; the second preset analysis sampling degree is greater than the first preset analysis sampling degree;
sending the second sampling analysis result to the database so that the database performs the execution plan optimization according to the second sampling analysis result;
re-executing the SQL sentence to be processed and recording a second execution time, and when the second execution time is determined to be larger than the preset time, performing third sampling analysis on the entity table according to the second preset analysis sampling degree, and performing histogram analysis on the entity table to obtain a third sampling analysis result and the data histogram;
and sending the third sampling analysis result and the data histogram to the database so that the database performs the execution plan optimization according to the third sampling analysis result and the data histogram.
6. The database optimization method according to claim 5, further comprising, after sending the third sample analysis result and the data histogram to the database:
re-executing the SQL sentence to be processed, recording third execution time, and judging whether the third execution time is greater than the preset time;
if yes, outputting preset alarm information.
7. The database optimization method according to claim 5, further comprising, after sending the third sample analysis result and the data histogram to the database:
setting the SQL statement to be processed into a processed SQL statement, and generating an optimization log for the processed SQL statement;
counting the number of optimized logs corresponding to the same processed SQL statement, and judging whether the number of the optimized logs is greater than a preset threshold value or not;
if yes, the processed SQL statement is reset into the SQL statement to be processed and sent to the database for execution, and the step of recording the execution time when the SQL statement to be processed is detected to be executed by the database is carried out.
8. A database optimization apparatus, comprising:
the acquisition module is used for acquiring the SQL sentences to be processed;
the execution time recording module is used for recording the execution time when the SQL sentence to be processed is detected to be executed by the database, and searching the entity table corresponding to the SQL sentence to be processed in the database when the execution time is determined to be greater than the preset time;
and the optimization analysis module is used for carrying out sampling analysis and/or histogram analysis on the entity table and sending the obtained sampling analysis result and/or data histogram to the database so that the database carries out plan optimization according to the sampling analysis and/or data histogram.
9. An electronic device, comprising:
a memory for storing a computer program;
a processor for implementing the database optimization method of any one of claims 1 to 7 when executing the computer program.
10. A storage medium having stored thereon computer-executable instructions which, when loaded and executed by a processor, carry out a method of database optimization according to any one of claims 1 to 7.
CN202210012709.8A 2022-01-06 2022-01-06 Database optimization method and device, electronic equipment and storage medium Pending CN114238395A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210012709.8A CN114238395A (en) 2022-01-06 2022-01-06 Database optimization method and device, electronic equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210012709.8A CN114238395A (en) 2022-01-06 2022-01-06 Database optimization method and device, electronic equipment and storage medium

Publications (1)

Publication Number Publication Date
CN114238395A true CN114238395A (en) 2022-03-25

Family

ID=80745949

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210012709.8A Pending CN114238395A (en) 2022-01-06 2022-01-06 Database optimization method and device, electronic equipment and storage medium

Country Status (1)

Country Link
CN (1) CN114238395A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116991691A (en) * 2023-09-26 2023-11-03 中国铁塔股份有限公司 Database testing method and device, electronic equipment and readable storage medium

Citations (22)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20040210563A1 (en) * 2003-04-21 2004-10-21 Oracle International Corporation Method and system of collecting execution statistics of query statements
US20050097091A1 (en) * 2003-09-06 2005-05-05 Oracle International Corporation SQL tuning base
US20060053164A1 (en) * 2004-09-03 2006-03-09 Teracruz, Inc. Application-layer monitoring of communication between one or more database clients and one or more database servers
US20060112093A1 (en) * 2004-11-22 2006-05-25 International Business Machines Corporation Method, system, and program for collecting statistics of data stored in a database
US20080133454A1 (en) * 2004-10-29 2008-06-05 International Business Machines Corporation System and method for updating database statistics according to query feedback
US20090077016A1 (en) * 2007-09-14 2009-03-19 Oracle International Corporation Fully automated sql tuning
US20090083215A1 (en) * 2007-09-21 2009-03-26 Louis Burger System, method, and computer-readable medium for automated selection of sampling usage in a database system
US20100005340A1 (en) * 2008-07-02 2010-01-07 Oracle International Corporation Test execution of user SQL in database server code
CN103098048A (en) * 2010-07-22 2013-05-08 惠普发展公司有限责任合伙企业 SQL enumerator
CN104620239A (en) * 2012-09-28 2015-05-13 甲骨文国际公司 Adaptive query optimization
US20150149441A1 (en) * 2013-11-25 2015-05-28 Anisoara Nica Data Statistics in Data Management Systems
CN105069134A (en) * 2015-08-18 2015-11-18 上海新炬网络信息技术有限公司 Method for automatically collecting Oracle statistical information
US20170031987A1 (en) * 2015-07-31 2017-02-02 International Business Machines Corporation Updating database statistics during query execution
CN107025224A (en) * 2016-01-29 2017-08-08 阿里巴巴集团控股有限公司 A kind of method and apparatus of monitor task operation
US20170286487A1 (en) * 2015-01-16 2017-10-05 International Business Machines Corporation Database statistical histogram forecasting
CN107609028A (en) * 2017-08-09 2018-01-19 山东中创软件商用中间件股份有限公司 A kind of determination method and device of inefficient SQL statement
CN108509530A (en) * 2018-03-14 2018-09-07 武汉斗鱼网络科技有限公司 A kind of slow query statement automatic optimization method of MySQL, computer equipment and storage medium
CN108664635A (en) * 2018-05-15 2018-10-16 上海达梦数据库有限公司 Acquisition methods, device, equipment and the storage medium of statistics of database information
CN110297814A (en) * 2019-05-22 2019-10-01 中国平安人寿保险股份有限公司 Method for monitoring performance, device, equipment and the storage medium of database manipulation
CN110555035A (en) * 2018-05-31 2019-12-10 阿里巴巴集团控股有限公司 Method and device for optimizing query statement
US20210406717A1 (en) * 2020-06-29 2021-12-30 Oracle International Corporation Enabling efficient machine learning model inference using adaptive sampling for autonomous database services
CN114270333A (en) * 2019-05-31 2022-04-01 微软技术许可有限责任公司 System and method for cardinality estimation feedback loop in query processing

Patent Citations (23)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20040210563A1 (en) * 2003-04-21 2004-10-21 Oracle International Corporation Method and system of collecting execution statistics of query statements
US20050097091A1 (en) * 2003-09-06 2005-05-05 Oracle International Corporation SQL tuning base
US20060053164A1 (en) * 2004-09-03 2006-03-09 Teracruz, Inc. Application-layer monitoring of communication between one or more database clients and one or more database servers
US20080133454A1 (en) * 2004-10-29 2008-06-05 International Business Machines Corporation System and method for updating database statistics according to query feedback
US20060112093A1 (en) * 2004-11-22 2006-05-25 International Business Machines Corporation Method, system, and program for collecting statistics of data stored in a database
US20090077016A1 (en) * 2007-09-14 2009-03-19 Oracle International Corporation Fully automated sql tuning
US20090083215A1 (en) * 2007-09-21 2009-03-26 Louis Burger System, method, and computer-readable medium for automated selection of sampling usage in a database system
US20100005340A1 (en) * 2008-07-02 2010-01-07 Oracle International Corporation Test execution of user SQL in database server code
CN103098048A (en) * 2010-07-22 2013-05-08 惠普发展公司有限责任合伙企业 SQL enumerator
CN104620239A (en) * 2012-09-28 2015-05-13 甲骨文国际公司 Adaptive query optimization
US20150149441A1 (en) * 2013-11-25 2015-05-28 Anisoara Nica Data Statistics in Data Management Systems
US20170286487A1 (en) * 2015-01-16 2017-10-05 International Business Machines Corporation Database statistical histogram forecasting
US20170031987A1 (en) * 2015-07-31 2017-02-02 International Business Machines Corporation Updating database statistics during query execution
CN105069134A (en) * 2015-08-18 2015-11-18 上海新炬网络信息技术有限公司 Method for automatically collecting Oracle statistical information
CN107025224A (en) * 2016-01-29 2017-08-08 阿里巴巴集团控股有限公司 A kind of method and apparatus of monitor task operation
CN107609028A (en) * 2017-08-09 2018-01-19 山东中创软件商用中间件股份有限公司 A kind of determination method and device of inefficient SQL statement
CN108509530A (en) * 2018-03-14 2018-09-07 武汉斗鱼网络科技有限公司 A kind of slow query statement automatic optimization method of MySQL, computer equipment and storage medium
CN108664635A (en) * 2018-05-15 2018-10-16 上海达梦数据库有限公司 Acquisition methods, device, equipment and the storage medium of statistics of database information
CN110555035A (en) * 2018-05-31 2019-12-10 阿里巴巴集团控股有限公司 Method and device for optimizing query statement
CN110297814A (en) * 2019-05-22 2019-10-01 中国平安人寿保险股份有限公司 Method for monitoring performance, device, equipment and the storage medium of database manipulation
CN114270333A (en) * 2019-05-31 2022-04-01 微软技术许可有限责任公司 System and method for cardinality estimation feedback loop in query processing
US20210406717A1 (en) * 2020-06-29 2021-12-30 Oracle International Corporation Enabling efficient machine learning model inference using adaptive sampling for autonomous database services
CN115735200A (en) * 2020-06-29 2023-03-03 甲骨文国际公司 Enabling efficient machine learning model reasoning using adaptive sampling for autonomous database services

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116991691A (en) * 2023-09-26 2023-11-03 中国铁塔股份有限公司 Database testing method and device, electronic equipment and readable storage medium
CN116991691B (en) * 2023-09-26 2024-01-30 中国铁塔股份有限公司 Database testing method and device, electronic equipment and readable storage medium

Similar Documents

Publication Publication Date Title
CN107609028B (en) Method and device for determining low-efficiency SQL statement
CN106997316B (en) System and method for detecting abnormal increase of memory
CN107678972B (en) Test case evaluation method and related device
CN110543356A (en) abnormal task detection method, device and equipment and computer storage medium
US20240036841A1 (en) Method and Apparatus for Compatibility Detection, Device and Non-transitory computer-readable storage medium
CN103842973A (en) Monitoring stored procedure execution
CN111427784B (en) Data acquisition method, device, equipment and storage medium
CN114238395A (en) Database optimization method and device, electronic equipment and storage medium
CN110377519B (en) Performance capacity test method, device and equipment of big data system and storage medium
CN109271453B (en) Method and device for determining database capacity
KR101830936B1 (en) Performance Improving System Based Web for Database and Application
KR20120087223A (en) System and method monitoring customer's complants
CN111767213B (en) Database check point testing method and device, electronic equipment and storage medium
CN110580170B (en) Method and device for identifying software performance risk
CN112948262A (en) System test method, device, computer equipment and storage medium
CN110716859A (en) Method for automatically pushing test cases for modified codes and related device
CN107342917B (en) Method and apparatus for detecting network device performance
CN114490413A (en) Test data preparation method and device, storage medium and electronic equipment
CN108763092B (en) Code defect detection method and device based on cross validation
CN112860469A (en) Method, device, equipment and storage medium for collecting information of katon log
CN112506565A (en) Method, medium, and apparatus for determining importance of processor micro-architectural event
CN112099838B (en) Method, device and storage medium for determining version difference
CN115629950B (en) Extraction method of performance test asynchronous request processing time point
CN113742216B (en) Method, device and storage medium for detecting efficiency of machine learning engine
CN113360362B (en) Dynamic sql efficiency checking method and plug-in

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination