CN117056307A - Database management method, apparatus, device, storage medium, and program product - Google Patents
Database management method, apparatus, device, storage medium, and program product Download PDFInfo
- Publication number
- CN117056307A CN117056307A CN202311007799.2A CN202311007799A CN117056307A CN 117056307 A CN117056307 A CN 117056307A CN 202311007799 A CN202311007799 A CN 202311007799A CN 117056307 A CN117056307 A CN 117056307A
- Authority
- CN
- China
- Prior art keywords
- access request
- processing
- resource
- condition
- difference
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000007726 management method Methods 0.000 title claims abstract description 40
- 238000003860 storage Methods 0.000 title claims abstract description 14
- 238000012545 processing Methods 0.000 claims abstract description 264
- 238000000034 method Methods 0.000 claims abstract description 84
- 238000001514 detection method Methods 0.000 claims abstract description 73
- 230000008569 process Effects 0.000 claims abstract description 32
- 238000004590 computer program Methods 0.000 claims abstract description 26
- 238000012549 training Methods 0.000 claims description 24
- 239000000725 suspension Substances 0.000 claims description 23
- 230000002159 abnormal effect Effects 0.000 claims description 8
- 238000013473 artificial intelligence Methods 0.000 abstract description 2
- 238000010586 diagram Methods 0.000 description 8
- 230000005856 abnormality Effects 0.000 description 6
- 230000000694 effects Effects 0.000 description 6
- 238000004891 communication Methods 0.000 description 3
- 238000012360 testing method Methods 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 238000004140 cleaning Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- OKTJSMMVPCPJKN-UHFFFAOYSA-N Carbon Chemical compound [C] OKTJSMMVPCPJKN-UHFFFAOYSA-N 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000003066 decision tree Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 229910021389 graphene Inorganic materials 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 238000011835 investigation Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000012706 support-vector machine Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/217—Database tuning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Computation (AREA)
- Medical Informatics (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present application relates to a database management method, apparatus, computer device, storage medium and computer program product. Relates to the technical field of artificial intelligence. The method comprises the following steps: processing the access request, and obtaining a first time length which is required to be consumed by normal processing of the access request and a second resource duty ratio which is required by normal processing according to the trained prediction model; in the process of processing the access request, detecting the processing condition of the access request, and updating the current detection times; judging whether the current detection times are greater than a first threshold value or not under the condition that the processing of the access request is not completed, and acquiring a time difference under the condition that the current detection times are not greater than the first threshold value; and executing the stopping operation of the processing procedure corresponding to the access request under the condition that the time difference is smaller than the first duration and the first resource difference is smaller than the preset resource difference. By adopting the method, the performance problem in the database can be avoided as much as possible.
Description
Technical Field
The present application relates to the field of artificial intelligence, and in particular, to a database management method, apparatus, computer device, storage medium, and computer program product.
Background
When a production application system is temporarily unavailable or unavailable for a long time, a large part of reasons are related to the performance of the database, and the database at a certain moment has performance problems due to the fact that a large number of users access the system simultaneously and the uncertainty of the access system, so that the whole application system is slow in response or the application system is unavailable.
Therefore, it is particularly important for management of database performance. In the traditional technology, only when the performance problem occurs in the database, the monitoring system gives an alarm, operation and maintenance personnel processes the database after receiving the alarm, and at the moment, a large number of SQL sentences can be found to be running, so that a database administrator is required to check each SQL sentence, and then the problematic SQL sentences are checked and killed.
However, the processing of the problems by the processing method has delay, so that the problems in the database can not be found timely, and the performance problems in the database can not be reduced.
Disclosure of Invention
In view of the foregoing, it is desirable to provide a database management method, apparatus, computer device, computer readable storage medium, and computer program product that can reduce the occurrence of performance problems in a database.
In a first aspect, the present application provides a database management method. The method comprises the following steps:
recording a first resource duty ratio when receiving an access request under the condition that the access request is received;
processing the access request, and obtaining a first time length which is required to be consumed by normal processing of the access request and a second resource duty ratio which is required by normal processing according to the trained prediction model;
in the process of processing the access request, detecting the processing condition of the access request according to a first time interval, and updating the current detection times;
judging whether the current detection times are larger than a first threshold value or not under the condition that the processing of the access request is not completed, and acquiring a time difference under the condition that the current detection times are not larger than the first threshold value; the time difference is the time difference from the start of executing the access request to the current moment;
and executing the suspension operation of the processing procedure corresponding to the access request under the condition that the time difference is smaller than the first duration and the first resource difference is smaller than a preset resource difference, wherein the first resource difference is obtained based on the first resource duty ratio, the second resource duty ratio and the current resource duty ratio.
In one embodiment, the method further comprises: and executing the suspension operation of the processing procedure corresponding to the access request under the condition that the current detection times are larger than a first threshold value.
In one embodiment, the method further comprises:
and returning to the step of detecting the processing condition of the access request according to a first time interval and continuously executing the step under the condition that the time difference is not smaller than the first time length or the condition that the time difference is smaller than the first time length and the first resource difference is not smaller than the preset resource difference.
In one embodiment, the obtaining, according to the trained prediction model, a first duration that needs to be consumed by normal processing of the access request and a second resource duty ratio that needs to be consumed by normal processing includes:
acquiring data information of the access request, wherein the data information comprises metadata information and SQL sentences corresponding to the access request;
and inputting the data information and the first resource duty ratio into a trained prediction model to obtain a first duration and a second resource duty ratio which are consumed by normally processing the access request.
In one embodiment, the method further comprises:
Acquiring feedback information, wherein the feedback information comprises an access request received in a preset time period and processing information of the access request; the processing information comprises a processing result, a processing duration and a resource duty ratio spent in processing; the processing results comprise normal processing results and abnormal processing results;
and updating the prediction model according to the feedback information.
In one embodiment, the method further comprises:
acquiring access log information; the access log information comprises execution related information of normally executed access requests and execution related information of abnormally executed access requests; the execution related information comprises data information of the access request, processing request duration and resource occupation ratio spent for processing the access request;
determining an access processing data sample according to the access log information;
and adjusting parameters of the pre-training prediction model according to the access processing sample until the pre-training prediction model converges to obtain a post-training prediction model.
In a second aspect, the application further provides a database management device. The device comprises:
the recording module is used for recording the first resource duty ratio when the access request is received under the condition that the access request is received;
The processing request and prediction module is used for processing the access request and obtaining a first time length which is required to be consumed by normal processing of the access request and a second resource duty ratio which is required by normal processing according to the trained prediction model;
the processing condition detection module is used for detecting the processing condition of the access request according to a first time interval in the process of processing the access request and updating the current detection times;
the time difference acquisition module is used for judging whether the current detection times are greater than a first threshold value or not under the condition that the processing condition of the access request is not completed, and acquiring the time difference under the condition that the current detection times are not greater than the first threshold value; the time difference is the time difference from the start of executing the access request to the current moment;
and the first suspension operation execution module is used for executing suspension operation of the processing procedure corresponding to the access request under the condition that the time difference is smaller than the first duration and the first resource difference is smaller than a preset resource difference, and the first resource difference is obtained based on the first resource ratio, the second resource ratio and the current resource ratio.
In a third aspect, the present application also provides a computer device. The computer device comprises a memory storing a computer program and a processor which when executing the computer program performs the steps of:
recording a first resource duty ratio when receiving an access request under the condition that the access request is received;
processing the access request, and obtaining a first time length which is required to be consumed by normal processing of the access request and a second resource duty ratio which is required by normal processing according to the trained prediction model;
in the process of processing the access request, detecting the processing condition of the access request according to a first time interval, and updating the current detection times;
judging whether the current detection times are larger than a first threshold value or not under the condition that the processing of the access request is not completed, and acquiring a time difference under the condition that the current detection times are not larger than the first threshold value; the time difference is the time difference from the start of executing the access request to the current moment;
and executing the suspension operation of the processing procedure corresponding to the access request under the condition that the time difference is smaller than the first duration and the first resource difference is smaller than a preset resource difference, wherein the first resource difference is obtained based on the first resource duty ratio, the second resource duty ratio and the current resource duty ratio.
In a fourth aspect, the present application also provides a computer-readable storage medium. The computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of:
recording a first resource duty ratio when receiving an access request under the condition that the access request is received;
processing the access request, and obtaining a first time length which is required to be consumed by normal processing of the access request and a second resource duty ratio which is required by normal processing according to the trained prediction model;
in the process of processing the access request, detecting the processing condition of the access request according to a first time interval, and updating the current detection times;
judging whether the current detection times are larger than a first threshold value or not under the condition that the processing of the access request is not completed, and acquiring a time difference under the condition that the current detection times are not larger than the first threshold value; the time difference is the time difference from the start of executing the access request to the current moment;
and executing the suspension operation of the processing procedure corresponding to the access request under the condition that the time difference is smaller than the first duration and the first resource difference is smaller than a preset resource difference, wherein the first resource difference is obtained based on the first resource duty ratio, the second resource duty ratio and the current resource duty ratio.
In a fifth aspect, the present application also provides a computer program product. The computer program product comprises a computer program which, when executed by a processor, implements the steps of:
recording a first resource duty ratio when receiving an access request under the condition that the access request is received;
processing the access request, and obtaining a first time length which is required to be consumed by normal processing of the access request and a second resource duty ratio which is required by normal processing according to the trained prediction model;
in the process of processing the access request, detecting the processing condition of the access request according to a first time interval, and updating the current detection times;
judging whether the current detection times are larger than a first threshold value or not under the condition that the processing of the access request is not completed, and acquiring a time difference under the condition that the current detection times are not larger than the first threshold value; the time difference is the time difference from the start of executing the access request to the current moment;
and executing the suspension operation of the processing procedure corresponding to the access request under the condition that the time difference is smaller than the first duration and the first resource difference is smaller than a preset resource difference, wherein the first resource difference is obtained based on the first resource duty ratio, the second resource duty ratio and the current resource duty ratio.
The database management method, apparatus, computer device, storage medium, and computer program product described above, in the case of receiving an access request, record a first resource duty ratio at the time of receiving the access request; processing the access request, and obtaining a first time length which is required to be consumed by normal processing of the access request and a second resource duty ratio which is required by normal processing according to the trained prediction model; in the process of processing the access request, detecting the processing condition of the access request according to a first time interval, and updating the current detection times; judging whether the current detection times are greater than a first threshold value or not under the condition that the processing of the access request is not completed, and acquiring a time difference under the condition that the current detection times are not greater than the first threshold value; under the condition that the time difference is smaller than the first duration and the first resource difference is smaller than the preset resource difference, the stopping operation of the processing process corresponding to the access request is executed, compared with the problem in the database which needs to be checked manually after an alarm occurs in the prior art, the method is adopted to realize the prediction of the time spent on normal processing of the access request and the resource occupation ratio through the prediction model, so that the processing process corresponding to the access request which possibly has abnormality is stopped according to the comparison of the prediction result and the actual processing condition of the access request, the performance problem in the database is avoided as much as possible, the performance of the database is maintained, and the management effect on the database is improved.
Drawings
FIG. 1 is a diagram of an application environment for a database management method in one embodiment;
FIG. 2 is a flow diagram of a database management method in one embodiment;
FIG. 3 is a flow diagram of training of a predictive model in one embodiment;
FIG. 4 is a flow chart of a database management method according to another embodiment;
FIG. 5 is a block diagram of a database management apparatus in one embodiment;
fig. 6 is an internal structural diagram of a computer device in one embodiment.
Detailed Description
The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.
The database management method provided by the embodiment of the application can be applied to an application environment shown in figure 1. Wherein the terminal 102 communicates with the server 104 via a network. The data storage system may store data that the server 104 needs to process. The data storage system may be integrated on the server 104 or may be located on a cloud or other network server.
The server 104 records a first resource duty ratio when receiving the access request in the case of receiving the access request of the terminal 102; processing the access request, and obtaining a first time length which is required to be consumed by normal processing of the access request and a second resource duty ratio which is required by normal processing according to the trained prediction model; in the process of processing the access request, detecting the processing condition of the access request according to a first time interval, and updating the current detection times; judging whether the current detection times are greater than a first threshold value or not under the condition that the processing of the access request is not completed, and acquiring a time difference under the condition that the current detection times are not greater than the first threshold value; and executing the stopping operation of the processing procedure corresponding to the access request under the condition that the time difference is smaller than the first duration and the first resource difference is smaller than the preset resource difference.
The terminal 102 may be, but not limited to, various personal computers, notebook computers, smart phones, tablet computers, internet of things devices, and portable wearable devices, where the internet of things devices may be smart speakers, smart televisions, smart air conditioners, smart vehicle devices, and the like. The portable wearable device may be a smart watch, smart bracelet, headset, or the like. The server 104 may be implemented as a stand-alone server or as a server cluster of multiple servers.
In one embodiment, as shown in fig. 2, a database management method is provided, and the method is applied to the server 104 in fig. 1 for illustration, and includes the following steps:
step 202, in the case of receiving an access request, recording a first resource duty ratio when receiving the access request.
Wherein the first resource duty cycle is a system performance condition when the access request is received.
For example, the server may record a first resource duty cycle when receiving the access request in the event that an access condition is received.
And 204, processing the access request, and obtaining a first time length required to be consumed by normal processing of the access request and a second resource duty ratio required by normal processing according to the trained prediction model.
The server establishes connection with the database to acquire data stored in the database through the server. The access request can comprise SQL sentences and is mainly divided into a plurality of types of adding, deleting, modifying and checking.
The prediction model may be a model obtained through deep learning training for making predictions of received access requests. The first duration is the time it takes to normally perform an access request in the case of current server system performance. The second resource duty cycle is the system resource that normally performs the access request under the current server system performance.
In some embodiments, the trained predictive model may be deployed to a server. The server pre-processes the data information of the access request to obtain data to be processed, predicts the data to be processed through a trained prediction model, and obtains a first time length required to be consumed by normal processing of the access request and a second resource duty ratio required by the normal processing. Specifically, preprocessing the data information of the access request may be cleaning the data information to remove abnormal data information and null data information in the data information, so as to obtain data to be processed.
For example, the server may process the access request, and obtain, while processing the access request, a first duration that is required to be consumed by normal processing of the access request and a second resource duty ratio required by normal processing according to the trained prediction model.
Step 206, in the process of processing the access request, detecting the processing condition of the access request according to the first time interval, and updating the current detection times.
The processing of the access request may be completed by execution or not. The current detection count is a count described for the current access request.
Specifically, the first time interval may be set based on actual conditions, and the present invention is not limited herein.
For example, in processing the access request, the server may detect the processing request of the access request at a first time interval, and update the current detection times.
Step 208, judging whether the current detection times are greater than a first threshold value or not when the processing condition of the access request is that the access request is not completed, and acquiring a time difference when the current detection times are not greater than the first threshold value; the time difference is the time difference from the start of executing the access request to the current moment.
Wherein the time difference is a time difference from the start of executing the access request to the current detection of whether the access request is processed.
Specifically, the magnitude of the first threshold may be set according to an empirical value, which is not limited herein.
For example, in the case where the processing request of the access request is not completed, the server may determine whether the current detection number is greater than the first threshold, and in the case where the current detection number is not greater than the first threshold, the server may acquire the time difference.
And step 210, executing a suspension operation of the processing procedure corresponding to the access request when the time difference is smaller than the first duration and the first resource difference is smaller than a preset resource difference, wherein the first resource difference is obtained based on the first resource duty ratio, the second resource duty ratio and the current resource duty ratio.
The first resource difference may be based on the first resource duty cycle, the second resource duty cycle, and the current resource duty cycle. Specifically, the first resource difference may be obtained by subtracting the current resource ratio from the sum of the first resource ratio and the second resource ratio.
For example, in the case where the time difference is smaller than the first duration and the first resource difference is smaller than the preset resource difference, the server may execute the suspension operation of the processing procedure corresponding to the access request. Specifically, the size of the first duration and the size of the preset resource difference may be empirical values, and may be set based on actual situations, which is not limited herein.
In the database management method, when an access request is received, a first resource duty ratio when the access request is received is recorded; processing the access request, and obtaining a first time length which is required to be consumed by normal processing of the access request and a second resource duty ratio which is required by normal processing according to the trained prediction model; in the process of processing the access request, detecting the processing condition of the access request according to a first time interval, and updating the current detection times; judging whether the current detection times are greater than a first threshold value or not under the condition that the processing of the access request is not completed, and acquiring a time difference under the condition that the current detection times are not greater than the first threshold value; under the condition that the time difference is smaller than the first duration and the first resource difference is smaller than the preset resource difference, the stopping operation of the processing process corresponding to the access request is executed, compared with the problem in the database which needs to be checked manually after an alarm occurs in the prior art, the method is adopted to realize the prediction of the time spent on normal processing of the access request and the resource occupation ratio through the prediction model, so that the processing process corresponding to the access request which possibly has abnormality is stopped according to the comparison of the prediction result and the actual processing condition of the access request, the performance problem in the database is avoided as much as possible, the performance of the database is maintained, and the management effect on the database is improved.
In one embodiment, the method further comprises:
and executing the suspension operation of the processing procedure corresponding to the access request under the condition that the current detection times are larger than a first threshold value.
For example, in the case where it is detected that the processing of the access request is incomplete, the current detection number is updated, and in the case where the current detection number is greater than the first threshold, the server may perform an abort operation of the processing procedure corresponding to the access request.
In the above embodiment, by setting the first threshold, the processing procedure corresponding to the access request with the possibility of abnormality is stopped, so that the performance problem in the database is avoided as much as possible, the performance of the database is maintained, and the management effect on the database is improved.
In one embodiment, the method further comprises:
and returning to the step of detecting the processing condition of the access request according to a first time interval and continuously executing the step under the condition that the time difference is not smaller than the first time length or the condition that the time difference is smaller than the first time length and the first resource difference is not smaller than the preset resource difference.
Wherein the time difference may be the time from the start of executing the access request to the current detection of whether the access process is completed. The first resource difference may be based on the first resource duty cycle, the second resource duty cycle, and the current resource duty cycle. Specifically, the first resource difference may be obtained by subtracting the current resource ratio from the sum of the first resource ratio and the second resource ratio.
Illustratively, it is determined whether the time difference is less than a first time period, and if the time difference is not less than the first time period, the step of detecting the processing condition of the access request at the first time interval is returned and the execution is continued. Or if the time difference is smaller than the first duration, judging whether the first resource difference is smaller than the preset resource difference, and if the first resource difference is not smaller than the preset resource difference, returning to the step of detecting the processing condition of the access request according to the first time interval and continuing to execute.
In the above embodiment, whether the processing condition of the access request needs to be continuously detected according to the first time interval is determined through the first time length and the preset resource difference value, so that the re-detection of the processing condition of the access condition is realized, the processing process corresponding to the access request with the possibility of abnormality is stopped, the performance problem in the database is avoided as much as possible, and the management effect of the database is improved.
In one embodiment, step 204 includes:
step 2042, obtaining data information of the access request, where the data information includes metadata information and an SQL statement corresponding to the access request.
And 2044, inputting the data information and the first resource ratio into a trained prediction model to obtain a first duration and a second resource ratio which are consumed for normally processing the access request.
The data information of the access request may include metadata information and an SQL statement corresponding to the access request. SQL statements are mainly divided into adding, deleting, modifying and looking up types.
The first resource duty cycle may be a system performance condition at the time of receiving the access request. Specifically, the usage resource ratio of the processor in the server may be the case when the access request is received.
The trained prediction model is obtained through deep learning training, and data information corresponding to the access request and the first resource ratio are input into the trained prediction model to obtain a first duration and a second resource ratio which are consumed for normally processing the access request.
In the above embodiment, the first duration and the second resource duty ratio, which are consumed for predicting normal processing of the access request, are performed through the trained prediction model, so that the processing process corresponding to the access request, which may have an abnormality, is stopped according to the comparison between the prediction result and the actual processing condition of the access request, so that performance problems in the database are avoided as much as possible, performance of the database is maintained, and management effect of the database is improved.
In one embodiment, the method further comprises:
acquiring feedback information, wherein the feedback information comprises an access request received in a preset time period and processing information of the access request; the processing information comprises a processing result, a processing duration and a resource duty ratio spent in processing; the processing results comprise normal processing results and abnormal processing results;
and updating the prediction model according to the feedback information.
The feedback information may include an access request received in a preset period of time and processing information of the access request. The processing information includes, but is not limited to, the processing result, the processing duration, and the resource duty cycle spent processing. The processing results include normal processing results and abnormal processing results. Specifically, the size of the preset time period may be selected based on practical situations, and the present invention is not limited herein.
For example, a preset period may be set, and feedback information is obtained from log information of the database according to the preset period, so as to update the trained prediction model according to the feedback information.
In the embodiment, the trained prediction model is updated through the feedback information, so that the real execution condition of the access request is realized, the prediction model is updated, and the accuracy of the first time length and the second resource occupation ratio which are consumed by using the prediction model to acquire the normal processing of the access request is improved.
In one embodiment, referring to FIG. 3, a flow diagram of training of a predictive model in one embodiment is shown, comprising:
step 302, access log information is obtained; the access log information comprises execution related information of normally executed access requests and execution related information of abnormally executed access requests; the execution related information includes data information of the access request, a processing request duration, and a resource duty ratio spent processing the access request.
The access log information is history information, and the access log information may include execution information for executing the access request. Specifically, the access log information may include execution-related information of an access request that is normally executed and execution-related information of an access request that is abnormally executed. The execution-related information may include data information of the corresponding access request, a processing request duration, and a resource duty ratio spent processing the request.
And step 304, determining access processing data samples according to the access log information.
For example, the access log information may be preprocessed to obtain data to be processed, so as to obtain access processing data samples according to the data to be processed. Specifically, the access log information may be subjected to data cleansing to remove abnormal data information and null data information in the access log information, so as to obtain an access processing data sample.
And 306, adjusting parameters of the pre-training prediction model according to the access processing sample until the pre-training prediction model converges to obtain a post-training prediction model.
In some embodiments, the server may use a portion of the access processing samples as a training set and the remaining portion as a test set, train the pre-training predictive model through the training set, and test the pre-training fault predictive model through the test set.
In practical application, the prediction model before training can be a decision tree model, a support vector machine model and a Bayesian model.
In the above embodiment, the history data, such as the access log information, is used to train the prediction model, so that accuracy of predicting the first duration consumed by normal processing of the access request and the second resource duty ratio required by normal processing by using the trained prediction model is improved.
For a better understanding of the complete process of database management in embodiments of the present invention, a complete example is described with reference to FIG. 4, which shows a schematic flow diagram of a database management method in another embodiment, comprising the steps of:
step 402, access log information is obtained, and data cleaning is performed on the access log information to obtain an access processing data sample.
The access log information comprises execution related information of normally executed access requests and execution related information of abnormally executed access requests; the execution related information includes data information of the access request, a processing request duration, and a resource duty ratio spent processing the access request.
And step 404, adjusting parameters of the pre-training prediction model according to the access processing sample until the pre-training prediction model converges to obtain a post-training prediction model.
In step 406, in the case of receiving the access request, the first resource duty cycle at the time of receiving the access request is recorded.
Step 408, processing the access request, obtaining data information of the access request, and inputting the data information and the first resource ratio into the trained prediction model to obtain the first duration and the second resource ratio which are required to be consumed for normally processing the access request.
The data information comprises metadata information and SQL sentences corresponding to the access requests.
In step 410, during the process of processing the access request, the processing condition of the access request is detected according to the first time interval, and the current detection times are updated.
Step 412, when the processing of the access request is not completed, determining whether the current detection number is greater than a first threshold, if the current detection number is not greater than the first threshold, acquiring a time difference, and if the time difference is less than a first duration and the first resource difference is less than a preset resource difference, executing a suspension operation of the processing procedure corresponding to the access request.
Specifically, the time difference is a time difference from the start of execution of the access request to the current time; specifically, the first resource difference is derived based on the first resource duty cycle, the second resource duty cycle, and the current resource duty cycle.
In step 414, if the current detection number is greater than the first threshold, an operation of suspending the processing procedure corresponding to the access request is performed.
In step 416, if the time difference is not less than the first duration, or if the time difference is less than the first duration and the first resource difference is not less than the preset resource difference, the step of detecting the processing condition of the access request according to the first time interval is returned and the execution is continued.
And 418, acquiring feedback information, and updating the prediction model according to the feedback information.
The feedback information comprises an access request received in a preset time period and processing information of the access request; the processing information comprises a processing result, a processing duration and a resource duty ratio spent by processing; the processing results include normal processing results and abnormal processing results.
In this embodiment, in the case of receiving an access request, a first resource duty ratio when the access request is received is recorded; processing the access request, and obtaining a first time length which is required to be consumed by normal processing of the access request and a second resource duty ratio which is required by normal processing according to the trained prediction model; in the process of processing the access request, detecting the processing condition of the access request according to a first time interval, and updating the current detection times; judging whether the current detection times are greater than a first threshold value or not under the condition that the processing of the access request is not completed, and acquiring a time difference under the condition that the current detection times are not greater than the first threshold value; under the condition that the time difference is smaller than the first duration and the first resource difference is smaller than the preset resource difference, the stopping operation of the processing process corresponding to the access request is executed, compared with the problem in the database which needs to be checked manually after an alarm occurs in the prior art, the method is adopted to realize the prediction of the time spent on normal processing of the access request and the resource occupation ratio through the prediction model, so that the processing process corresponding to the access request which possibly has abnormality is stopped according to the comparison of the prediction result and the actual processing condition of the access request, the performance problem in the database is avoided as much as possible, the performance of the database is maintained, and the management effect on the database is improved. Compared with the traditional mode that manual one-to-one investigation is needed, the method reduces labor cost.
It should be understood that, although the steps in the flowcharts related to the embodiments described above are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.
Based on the same inventive concept, the embodiment of the application also provides a database management device for realizing the above related database management method. The implementation of the solution provided by the device is similar to the implementation described in the above method, so the specific limitation in one or more embodiments of the database management device provided below may refer to the limitation of the database management method hereinabove, and will not be repeated herein.
In one embodiment, as shown in fig. 5, there is provided a database management apparatus including: a recording module 502, a processing request and prediction module 504, a processing situation detection module 506, a time difference acquisition module 508, and a first suspension operation execution module 510, wherein:
a recording module 502, configured to record, when an access request is received, a first resource duty ratio when the access request is received;
a processing request and prediction module 504, configured to process the access request, and obtain, according to the trained prediction model, a first duration that needs to be consumed for normal processing of the access request and a second resource duty ratio that needs to be consumed for normal processing;
a processing condition detection module 506, configured to detect, during processing the access request, a processing condition of the access request according to a first time interval, and update a current detection number;
a time difference obtaining module 508, configured to determine whether the current detection number is greater than a first threshold when the processing of the access request is not completed, and obtain a time difference when the current detection number is not greater than the first threshold; the time difference is the time difference from the start of executing the access request to the current moment;
And a first suspension operation execution module 510, configured to execute a suspension operation of a processing procedure corresponding to the access request when the time difference is smaller than the first duration and the first resource difference is smaller than a preset resource difference, where the first resource difference is obtained based on the first resource duty ratio, the second resource duty ratio, and the current resource duty ratio.
In some embodiments, the database management apparatus further comprises:
and the second suspension operation module is used for executing suspension operation of the processing procedure corresponding to the access request under the condition that the current detection times are larger than the first threshold value.
In some embodiments, the database management apparatus further comprises:
and the detection processing request module is used for returning to the step of detecting the processing condition of the access request according to a first time interval and continuously executing the processing condition under the condition that the time difference is not smaller than the first time length or the condition that the time difference is smaller than the first time length and the first resource difference is not smaller than the preset resource difference.
In some embodiments, the processing request and prediction module 504 includes:
the data information acquisition unit is used for acquiring data information of the access request, wherein the data information comprises metadata information and SQL sentences corresponding to the access request;
The model prediction unit is used for inputting the data information and the first resource duty ratio into a trained prediction model to obtain a first duration and a second resource duty ratio which are consumed for normally processing the access request.
In some embodiments, the database management apparatus further comprises:
the feedback information acquisition module is used for acquiring feedback information, wherein the feedback information comprises an access request received in a preset time period and processing information of the access request; the processing information comprises a processing result, a processing duration and a resource duty ratio spent in processing; the processing results comprise normal processing results and abnormal processing results;
and the prediction model updating module is used for updating the prediction model according to the feedback information.
In some embodiments, the database management apparatus further comprises:
the log information acquisition module is used for acquiring access log information; the access log information comprises execution related information of normally executed access requests and execution related information of abnormally executed access requests; the execution related information comprises data information of the access request, processing request duration and resource occupation ratio spent for processing the access request;
The data sample determining module is used for determining access processing data samples according to the access log information;
and the prediction model obtaining module is used for adjusting parameters of the prediction model before training according to the access processing sample until the prediction model before training converges to obtain the prediction model after training.
The various modules in the database management apparatus described above may be implemented in whole or in part by software, hardware, or a combination thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
In one embodiment, a computer device is provided, which may be a server, the internal structure of which may be as shown in fig. 6. The computer device includes a processor, a memory, an Input/Output interface (I/O) and a communication interface. The processor, the memory and the input/output interface are connected through a system bus, and the communication interface is connected to the system bus through the input/output interface. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device is for storing access log information. The input/output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a database management method. The display unit of the computer device is used for forming a visual picture, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, can also be a key, a track ball or a touch pad arranged on the shell of the computer equipment, and can also be an external keyboard, a touch pad or a mouse and the like.
It will be appreciated by those skilled in the art that the structure shown in FIG. 6 is merely a block diagram of some of the structures associated with the present inventive arrangements and is not limiting of the computer device to which the present inventive arrangements may be applied, and that a particular computer device may include more or fewer components than shown, or may combine some of the components, or have a different arrangement of components.
In one embodiment, a computer device is provided comprising a memory and a processor, the memory having stored therein a computer program, the processor when executing the computer program performing the steps of:
recording a first resource duty ratio when receiving an access request under the condition that the access request is received;
processing the access request, and obtaining a first time length which is required to be consumed by normal processing of the access request and a second resource duty ratio which is required by normal processing according to the trained prediction model;
in the process of processing the access request, detecting the processing condition of the access request according to a first time interval, and updating the current detection times;
judging whether the current detection times are larger than a first threshold value or not under the condition that the processing of the access request is not completed, and acquiring a time difference under the condition that the current detection times are not larger than the first threshold value; the time difference is the time difference from the start of executing the access request to the current moment;
And executing the suspension operation of the processing procedure corresponding to the access request under the condition that the time difference is smaller than the first duration and the first resource difference is smaller than a preset resource difference, wherein the first resource difference is obtained based on the first resource duty ratio, the second resource duty ratio and the current resource duty ratio.
In one embodiment, a computer readable storage medium is provided having a computer program stored thereon, which when executed by a processor, performs the steps of:
recording a first resource duty ratio when receiving an access request under the condition that the access request is received;
processing the access request, and obtaining a first time length which is required to be consumed by normal processing of the access request and a second resource duty ratio which is required by normal processing according to the trained prediction model;
in the process of processing the access request, detecting the processing condition of the access request according to a first time interval, and updating the current detection times;
judging whether the current detection times are larger than a first threshold value or not under the condition that the processing of the access request is not completed, and acquiring a time difference under the condition that the current detection times are not larger than the first threshold value; the time difference is the time difference from the start of executing the access request to the current moment;
And executing the suspension operation of the processing procedure corresponding to the access request under the condition that the time difference is smaller than the first duration and the first resource difference is smaller than a preset resource difference, wherein the first resource difference is obtained based on the first resource duty ratio, the second resource duty ratio and the current resource duty ratio.
In one embodiment, a computer program product is provided comprising a computer program which, when executed by a processor, performs the steps of:
recording a first resource duty ratio when receiving an access request under the condition that the access request is received;
processing the access request, and obtaining a first time length which is required to be consumed by normal processing of the access request and a second resource duty ratio which is required by normal processing according to the trained prediction model;
in the process of processing the access request, detecting the processing condition of the access request according to a first time interval, and updating the current detection times;
judging whether the current detection times are larger than a first threshold value or not under the condition that the processing of the access request is not completed, and acquiring a time difference under the condition that the current detection times are not larger than the first threshold value; the time difference is the time difference from the start of executing the access request to the current moment;
And executing the suspension operation of the processing procedure corresponding to the access request under the condition that the time difference is smaller than the first duration and the first resource difference is smaller than a preset resource difference, wherein the first resource difference is obtained based on the first resource duty ratio, the second resource duty ratio and the current resource duty ratio.
It should be noted that, the user information (including but not limited to user equipment information, user personal information, etc.) and the data (including but not limited to data for analysis, stored data, presented data, etc.) related to the present application are information and data authorized by the user or sufficiently authorized by each party, and the collection, use and processing of the related data need to comply with the related laws and regulations and standards of the related country and region.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high density embedded nonvolatile Memory, resistive random access Memory (ReRAM), magnetic random access Memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric Memory (Ferroelectric Random Access Memory, FRAM), phase change Memory (Phase Change Memory, PCM), graphene Memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in the form of a variety of forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), and the like. The databases referred to in the embodiments provided herein may include at least one of a relational database and a non-relational database. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processor referred to in the embodiments provided in the present application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, or the like, but is not limited thereto.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The foregoing examples illustrate only a few embodiments of the application and are described in detail herein without thereby limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of the application should be assessed as that of the appended claims.
Claims (10)
1. A method of database management, the method comprising:
recording a first resource duty ratio when receiving an access request under the condition that the access request is received;
processing the access request, and obtaining a first time length which is required to be consumed by normal processing of the access request and a second resource duty ratio which is required by normal processing according to the trained prediction model;
In the process of processing the access request, detecting the processing condition of the access request according to a first time interval, and updating the current detection times;
judging whether the current detection times are larger than a first threshold value or not under the condition that the processing of the access request is not completed, and acquiring a time difference under the condition that the current detection times are not larger than the first threshold value; the time difference is the time difference from the start of executing the access request to the current moment;
and executing the suspension operation of the processing procedure corresponding to the access request under the condition that the time difference is smaller than the first duration and the first resource difference is smaller than a preset resource difference, wherein the first resource difference is obtained based on the first resource duty ratio, the second resource duty ratio and the current resource duty ratio.
2. The method according to claim 1, wherein the method further comprises:
and executing the suspension operation of the processing procedure corresponding to the access request under the condition that the current detection times are larger than a first threshold value.
3. The method according to claim 1, wherein the method further comprises:
and returning to the step of detecting the processing condition of the access request according to a first time interval and continuously executing the step under the condition that the time difference is not smaller than the first time length or the condition that the time difference is smaller than the first time length and the first resource difference is not smaller than the preset resource difference.
4. The method according to claim 1, wherein obtaining a first duration that is required to be consumed by normal processing of the access request and a second resource duty ratio that is required by normal processing according to the trained predictive model includes:
acquiring data information of the access request, wherein the data information comprises metadata information and SQL sentences corresponding to the access request;
and inputting the data information and the first resource duty ratio into a trained prediction model to obtain a first duration and a second resource duty ratio which are consumed by normally processing the access request.
5. The method according to claim 1, wherein the method further comprises:
acquiring feedback information, wherein the feedback information comprises an access request received in a preset time period and processing information of the access request; the processing information comprises a processing result, a processing duration and a resource duty ratio spent in processing; the processing results comprise normal processing results and abnormal processing results;
and updating the prediction model according to the feedback information.
6. The method according to claim 1, wherein the method further comprises:
acquiring access log information; the access log information comprises execution related information of normally executed access requests and execution related information of abnormally executed access requests; the execution related information comprises data information of the access request, processing request duration and resource occupation ratio spent for processing the access request;
Determining an access processing data sample according to the access log information;
and adjusting parameters of the pre-training prediction model according to the access processing sample until the pre-training prediction model converges to obtain a post-training prediction model.
7. A database management apparatus, the apparatus comprising:
the recording module is used for recording the first resource duty ratio when the access request is received under the condition that the access request is received;
the processing request and prediction module is used for processing the access request and obtaining a first time length which is required to be consumed by normal processing of the access request and a second resource duty ratio which is required by normal processing according to the trained prediction model;
the processing condition detection module is used for detecting the processing condition of the access request according to a first time interval in the process of processing the access request and updating the current detection times;
the time difference acquisition module is used for judging whether the current detection times are greater than a first threshold value or not under the condition that the processing condition of the access request is not completed, and acquiring the time difference under the condition that the current detection times are not greater than the first threshold value; the time difference is the time difference from the start of executing the access request to the current moment;
And the first suspension operation execution module is used for executing suspension operation of the processing procedure corresponding to the access request under the condition that the time difference is smaller than the first duration and the first resource difference is smaller than a preset resource difference, and the first resource difference is obtained based on the first resource ratio, the second resource ratio and the current resource ratio.
8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the database management method of any of claims 1 to 6 when the computer program is executed.
9. A computer readable storage medium having stored thereon a computer program, characterized in that the computer program, when executed by a processor, implements the steps of the database management method of any of claims 1 to 6.
10. A computer program product comprising a computer program, characterized in that the computer program, when being executed by a processor, implements the steps of the database management method according to any of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311007799.2A CN117056307A (en) | 2023-08-10 | 2023-08-10 | Database management method, apparatus, device, storage medium, and program product |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311007799.2A CN117056307A (en) | 2023-08-10 | 2023-08-10 | Database management method, apparatus, device, storage medium, and program product |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117056307A true CN117056307A (en) | 2023-11-14 |
Family
ID=88654550
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311007799.2A Pending CN117056307A (en) | 2023-08-10 | 2023-08-10 | Database management method, apparatus, device, storage medium, and program product |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117056307A (en) |
-
2023
- 2023-08-10 CN CN202311007799.2A patent/CN117056307A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP6393805B2 (en) | Efficient query processing using histograms in the columnar database | |
| CN113946986A (en) | Method and device for evaluating average time before product failure based on accelerated degradation test | |
| CN116701033A (en) | Host switching abnormality detection method, device, computer equipment and storage medium | |
| CN111078500A (en) | Method and device for adjusting operation configuration parameters, computer equipment and storage medium | |
| CN113254153B (en) | Method and device for processing flow task, computer equipment and storage medium | |
| CN113946983A (en) | Method and device for evaluating weak links of product reliability and computer equipment | |
| CN117033039A (en) | Fault detection method, device, computer equipment and storage medium | |
| CN112379913A (en) | Software optimization method, device, equipment and storage medium based on risk identification | |
| CN116737495A (en) | Method, device, computer equipment and storage medium for determining running state | |
| CN115801541A (en) | Slow access warning method and device in full-link tracking platform and computer equipment | |
| CN117056307A (en) | Database management method, apparatus, device, storage medium, and program product | |
| CN115169155A (en) | Engine fault prediction method and device, computer equipment and storage medium | |
| CN112395167A (en) | Operation fault prediction method and device and electronic equipment | |
| CN116227240B (en) | Product life evaluation method, device and equipment based on comprehensive stress acceleration test | |
| CN116795705A (en) | Abnormal node determination method and device and computer equipment | |
| CN117389840A (en) | Database running state monitoring method and device and computer equipment | |
| CN116577586A (en) | Power equipment fault monitoring method, device, computer equipment and storage medium | |
| CN117435492A (en) | Database performance test method, device, computer equipment and storage medium | |
| CN115098331A (en) | IO (input/output) blocking detection method and device, computer equipment, storage medium and product | |
| CN117151873A (en) | Abnormality prompting method, abnormality prompting device, computer equipment and storage medium | |
| CN116664295A (en) | Transaction data processing method, apparatus, device, storage medium and program product | |
| CN117407761A (en) | Resource transfer method, apparatus, device, medium and program product | |
| CN116346671A (en) | Platform connection testing method, device, computer equipment and storage medium | |
| CN116681203A (en) | Enterprise management consultation method and system based on big data analysis | |
| CN116881092A (en) | Method, device, equipment and storage medium for predicting alarm data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |