CN114116835A - Query timeout prediction method and device - Google Patents
Query timeout prediction method and device Download PDFInfo
- Publication number
- CN114116835A CN114116835A CN202111339471.1A CN202111339471A CN114116835A CN 114116835 A CN114116835 A CN 114116835A CN 202111339471 A CN202111339471 A CN 202111339471A CN 114116835 A CN114116835 A CN 114116835A
- Authority
- CN
- China
- Prior art keywords
- time
- query
- data table
- target data
- expected
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 40
- 238000012549 training Methods 0.000 claims description 16
- 238000003062 neural network model Methods 0.000 claims description 5
- 238000012545 processing Methods 0.000 claims description 5
- 238000005457 optimization Methods 0.000 abstract description 4
- 230000000875 corresponding effect Effects 0.000 description 24
- 238000010801 machine learning Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000002596 correlated effect Effects 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2465—Query processing support for facilitating data mining operations in structured databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/26—Visual data mining; Browsing structured data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Computing Systems (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a query timeout prediction method and a query timeout prediction device, which can be applied to the distributed field or the financial field, and can obtain a query time fitting curve representing the corresponding relation between query data volume and query time by performing linear fitting on historical data, and accurately determine the estimated query time corresponding to the data volume of a target data table at the estimated filing time according to the fitting curve, so as to judge whether the target data table has query timeout at the estimated filing time based on the estimated query time, further facilitate the adjustment of the filing time or the optimization of the storage of the target data table under the condition of judging that the target data table has query timeout at the estimated filing time, avoid the occurrence of insufficient storage space error reporting or query timeout error reporting due to the overlarge data volume of the target data table, and improve the query efficiency.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a query timeout prediction method and a query timeout prediction device.
Background
With the development of information technology, various organizations generally adopt digital offices, and relevant data are stored in a system database.
Because new data can be generated every moment in the system, the data amount in the system database is continuously increased, so that the shortage of the system storage space or slow data query can be caused, and even the shortage of the system storage space or the overtime query error can be caused.
Disclosure of Invention
In view of this, the present invention provides a query timeout predicting method and apparatus, which accurately determine whether a query timeout exists in a target data table at a predicted archiving time.
In order to achieve the above purpose, the invention provides the following specific technical scheme:
a query timeout prediction method, comprising:
acquiring the data volume of the target data table in the expected filing time;
performing linear fitting on the historical data to obtain a query time fitting curve representing the corresponding relation between the query data amount and the query time;
according to the query time fitting curve, determining estimated query time corresponding to the data volume of the target data table at the estimated filing time;
and judging whether the target data table has query timeout at the expected filing time or not based on the estimated query time.
Optionally, the obtaining the data size when the target data table is archived includes:
acquiring the expected filing time of the target data table;
and predicting the data amount of the target data table at the expected filing time according to the historical data.
Optionally, obtaining the expected archiving time of the target data table includes:
analyzing the filing rule of the target data table in historical data;
and determining the expected filing time of the target data table according to the filing rule of the target data table.
Optionally, the predicting, according to the historical data, the data amount of the target data table at the expected filing time includes:
performing linear fitting on the data quantity of the target data table at different time in the historical data to obtain a data quantity fitting curve representing the corresponding relation between the data quantity and the time;
and according to the data volume fitting curve, predicting the data volume of the target data table at the expected filing time.
Optionally, the predicting, according to the historical data, the data amount of the target data table at the expected filing time includes:
acquiring time characteristics of expected filing time;
inputting the time characteristics of the expected filing time into a pre-constructed data quantity prediction model for processing to obtain the data quantity of the target data table at the expected filing time, wherein the data quantity prediction model is obtained by training and verifying a neural network model by using a training sample marked with the data quantity in advance, and the training sample is the data quantity sample of the target data table under different time characteristics.
Optionally, the method further includes:
and when the target data table is judged to have query timeout in the expected archiving time, the archiving time of the target data table is adjusted, and the adjusted archiving time is made to be before the expected archiving time.
Optionally, the method further includes:
and under the condition that the query timeout of the target data table in the expected filing time is judged, performing distributed storage on the target data table according to a preset rule, and modifying the query route of the target data table.
Optionally, after obtaining the data amount of the target data table at the expected archiving time, the method further includes:
judging whether the data volume of the target data table in the expected filing time exceeds the maximum data volume;
if the maximum data volume is exceeded, prompting that the storage space of the target data table in the expected filing time is insufficient;
and if the maximum data volume is not exceeded, performing linear fitting on the historical data to obtain a query time fitting curve representing the corresponding relation between the query data volume and the query time.
A query timeout predicting device, comprising:
the data volume acquisition unit is used for acquiring the data volume of the target data table at the expected filing time;
the linear fitting unit is used for performing linear fitting on the historical data to obtain a query time fitting curve representing the corresponding relation between the query data amount and the query time;
the query time prediction unit is used for determining the estimated query time corresponding to the data volume of the target data table at the estimated filing time according to the query time fitting curve;
and the query timeout judging unit is used for judging whether the target data table has query timeout at the expected filing time or not based on the estimated query time.
Optionally, the data amount obtaining unit includes:
the filing time acquiring subunit is used for acquiring the expected filing time of the target data table;
and the data quantity prediction subunit is used for predicting the data quantity of the target data table at the expected filing time according to the historical data.
Optionally, the filing time obtaining subunit is specifically configured to:
analyzing the filing rule of the target data table in historical data;
and determining the expected filing time of the target data table according to the filing rule of the target data table.
Optionally, the data amount predictor is specifically configured to:
performing linear fitting on the data quantity of the target data table at different time in the historical data to obtain a data quantity fitting curve representing the corresponding relation between the data quantity and the time;
and according to the data volume fitting curve, predicting the data volume of the target data table at the expected filing time.
Optionally, the data amount predictor is specifically configured to:
acquiring time characteristics of expected filing time;
inputting the time characteristics of the expected filing time into a pre-constructed data quantity prediction model for processing to obtain the data quantity of the target data table at the expected filing time, wherein the data quantity prediction model is obtained by training and verifying a neural network model by using a training sample marked with the data quantity in advance, and the training sample is the data quantity sample of the target data table under different time characteristics.
Optionally, the apparatus further comprises:
and the filing time adjusting unit is used for adjusting the filing time of the target data table to enable the adjusted filing time to be before the expected filing time when judging that the target data table has query timeout in the expected filing time.
Optionally, the apparatus further comprises:
and the storage optimization unit is used for performing distributed storage on the target data table according to a preset rule and modifying the query route of the target data table under the condition that the query timeout of the target data table in the expected filing time is judged.
Optionally, the apparatus further comprises:
the storage space judging unit is used for judging whether the data volume of the target data table in the expected filing time exceeds the maximum data volume; if the maximum data volume is exceeded, prompting that the storage space of the target data table in the expected filing time is insufficient; and if the maximum data volume is not exceeded, triggering the linear fitting unit.
Compared with the prior art, the invention has the following beneficial effects:
the invention discloses a query timeout prediction method, which obtains a query time fitting curve representing the corresponding relation between query data volume and query time by linear fitting of historical data, and can accurately determine the estimated query time corresponding to the data volume of a target data table at the estimated filing time according to the fitting curve, thereby judging whether the target data table has query timeout at the estimated filing time based on the estimated query time, further facilitating to adjust the filing time or optimize the storage of the target data table as early as possible under the condition that the target data table has query timeout at the estimated filing time, avoiding the occurrence of insufficient system storage space error reporting or query timeout error reporting due to overlarge data volume of the target data table, and improving the query efficiency.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
Fig. 1 is a schematic flowchart of a query timeout prediction method according to an embodiment of the present invention;
FIG. 2 is a flowchart illustrating another query timeout prediction method according to an embodiment of the present invention;
FIG. 3 is a flowchart illustrating another query timeout prediction method according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a query timeout predicting device according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The inventor finds out through research that: some systems, such as bank systems, have a data table whose data amount is increasing continuously, but because data is important, it needs to be queried periodically or aperiodically, data cannot be archived frequently, data needs to be archived as late as possible, but query timeout or insufficient storage space cannot be caused by too large data amount. Therefore, there is a need to accurately predict whether the data table will have a query timeout at the expected archive time.
On the basis, the invention provides a query timeout predicting method and a query timeout predicting device, which can accurately predict whether the data table has query timeout in the expected filing time, so that the filing time can be adjusted or the storage of the target data table can be optimized as early as possible under the condition that the query timeout in the expected filing time of the target data table is judged, the occurrence of system storage space deficiency error reporting or query timeout error reporting caused by overlarge data amount of the target data table is avoided, and the query efficiency is improved.
Specifically, referring to fig. 1, the present embodiment discloses a query timeout prediction method, which includes the following steps:
s101: acquiring the data volume of the target data table in the expected filing time;
the target data table is a centralized storage, in particular to a data table which is important, needs to be inquired regularly or irregularly and cannot be filed frequently.
The expected filing time may be estimated by a person or may be predicted based on historical data.
The method comprises the steps of acquiring the data volume of a target data table at the expected filing time, firstly acquiring the expected filing time of the target data table, and then predicting the data volume of the target data table at the expected filing time according to historical data.
The expected filing time of the target data table can be obtained in various ways, such as obtaining the expected filing time of the target data table which is manually input, wherein the expected filing time is estimated by experts according to experience, and can also be predicted by a computer according to historical data.
There may be various ways to predict the expected archival time of the target data table based on historical data, as illustrated by an example:
analyzing the filing rule of a target data table in historical data, such as analyzing the filing time of different data tables in the historical data, predicting the filing rule of the target data table by means of a machine learning model, wherein the machine learning model is obtained by historical data training, the historical data is the pre-collected historical filing time of different data tables, and after the filing rule of the target data table is determined, determining the predicted filing time of the target data table according to the filing rule of the target data table.
After determining the expected archiving time of the target data table, predicting the data amount of the target data table at the expected archiving time according to the historical data, which may be implemented in various ways, and the following is exemplified by two examples:
example 1
First, in this embodiment, it is assumed that the data volume of the target data table is mainly related to time, and on this basis, linear fitting is performed on the data volumes of the target data table at different times in the historical data to obtain a data volume fitting curve representing the correspondence between the data volume and the time.
Then, the data amount of the target data table at the expected filing time can be predicted by fitting a curve according to the data amount.
Example two
First, a time characteristic of the expected filing time is acquired, the time characteristic is correlated with a data amount growth characteristic of the target data table, and the time characteristic of the expected filing time may be set in advance according to the data amount growth characteristic of the target data table. For example: if the target data table increases the data amount faster than the data amount of the non-working day in each working day, the time characteristic of the expected filing time should include whether the target data table is a working day, and if the target data table increases the data amount faster in holidays or e-commerce shopping festivals, the time characteristic of the expected filing time should include whether the target data table is a holiday or e-commerce shopping festival.
Inputting the time characteristics of the expected filing time into a pre-constructed data quantity prediction model for processing to obtain the data quantity of the target data table at the expected filing time, wherein the data quantity prediction model is obtained by training and verifying a neural network model by using a training sample marked with the data quantity in advance, and the training sample is a data quantity sample of the target data table under different time characteristics.
It should be noted that the above two examples are only optional embodiments for predicting the data amount of the target data table at the expected archiving time according to the historical data, and the invention is not limited thereto.
S102: performing linear fitting on the historical data to obtain a query time fitting curve representing the corresponding relation between the query data amount and the query time;
specifically, in this embodiment, under the condition that the hardware and system resources are assumed to be stable, on the basis that the main influence factor of the query time is the query data volume, the query times corresponding to different query data volumes in the historical data are linearly fitted to obtain a query time fitting curve representing the correspondence between the query data volume and the query time.
S103: according to the query time fitting curve, determining the estimated query time corresponding to the data volume of the target data table at the estimated filing time;
and determining the estimated query time corresponding to the data quantity of the target data table at the estimated filing time according to the corresponding relation between the query data quantity of the target data table in the query time fitting curve and the query time, wherein the query time is estimated.
S104: and judging whether the target data table has query timeout at the expected filing time or not based on the estimated query time.
Specifically, by comparing the estimated query time with the longest query time preset by the system, when the estimated query time is greater than the longest query time preset by the system, it is determined that the target data table has query timeout in the estimated filing time, and otherwise, it is determined that the target data table does not have query timeout in the estimated filing time.
It should be noted that, if there is no query timeout in the target data table during the expected archiving time, the target data table may be archived according to the expected archiving time.
Therefore, according to the query timeout prediction method disclosed in this embodiment, a query time fitting curve representing the correspondence between the query data amount and the query time is obtained by performing linear fitting on the historical data, and the estimated query time corresponding to the data amount of the target data table at the expected filing time can be accurately determined according to the fitting curve, so that whether query timeout exists in the expected filing time of the target data table is accurately determined based on the estimated query time.
Further, under the condition that the query timeout exists in the expected filing time of the target data table, the filing time can be adjusted as early as possible or the storage of the target data table can be optimized, so that the situation that the system storage space is insufficient and the query timeout error occurs due to the fact that the data volume of the target data table is too large is avoided, and the query efficiency is improved.
Referring to fig. 2, an embodiment of the present invention discloses a query timeout prediction method, which specifically includes the following steps:
s201: acquiring the data volume of the target data table in the expected filing time;
s202: performing linear fitting on the historical data to obtain a query time fitting curve representing the corresponding relation between the query data amount and the query time;
s203: according to the query time fitting curve, determining the estimated query time corresponding to the data volume of the target data table at the estimated filing time;
s204: judging whether the target data table has query overtime in the expected filing time or not based on the estimated query time;
s205: and when the target data table is judged to have query timeout in the expected filing time, adjusting the filing time of the target data table to enable the adjusted filing time to be before the expected filing time.
Please refer to S101 to S104 in the above embodiments, and details of the implementation manners of S201 to S204 are not described herein.
Specifically, the archive time of the target data table may be adjusted according to a preset step size, so that the adjusted archive time is before the expected archive time.
After the filing time of the target data table is adjusted, the data volume of the target data table in the adjusted predicted filing time is obtained, then the adjusted predicted inquiry time corresponding to the data volume of the target data table in the adjusted predicted filing time is determined according to an inquiry time fitting curve, whether the adjusted predicted filing time of the target data table is overtime or not is judged based on the adjusted predicted inquiry time, if the adjusted predicted filing time is not overtime, the target data table is filed according to the adjusted predicted filing time, and if the adjusted predicted filing time is still overtime, the filing time of the target data table is continuously adjusted according to the preset step length until the adjusted predicted filing time of the target data table is not overtime.
Referring to fig. 3, an embodiment of the present invention discloses a query timeout prediction method, which specifically includes the following steps:
s301: acquiring the data volume of the target data table in the expected filing time;
s302: performing linear fitting on the historical data to obtain a query time fitting curve representing the corresponding relation between the query data amount and the query time;
s303: according to the query time fitting curve, determining the estimated query time corresponding to the data volume of the target data table at the estimated filing time;
s304: judging whether the target data table has query overtime in the expected filing time or not based on the estimated query time;
s305: and under the condition that the query timeout of the target data table in the expected filing time is judged, performing distributed storage on the target data table according to a preset rule, and modifying the query route of the target data table.
Please refer to S101 to S104 in the above embodiments, and detailed descriptions thereof are omitted here.
The preset rule for the distributed storage of the target data table is set by using an expert system, and can also be set manually by an expert.
Specifically, other distributed storage spaces in the intranet can be queried, and the target data table can be stored in the storage spaces in a distributed manner, so that the target data table is ensured to have enough storage space.
It should be noted that, because the storage mode of the target data table is changed, the query route of the target data table needs to be modified according to the distributed storage mode of the target data table, so that the modification of the storage mode of the target data table is not sensible to the user, and the user normally accesses the target data table according to the modified query route.
Further, the data size of the target data table at the expected filing time may exceed the maximum data size, which may cause a problem of insufficient storage space of the target data table.
On this basis, in this embodiment, after the data volume of the target data table at the expected filing time is obtained, it is determined whether the data volume of the target data table at the expected filing time exceeds the maximum data volume, if the data volume exceeds the maximum data volume, it is prompted that the storage space of the target data table at the expected filing time is insufficient, and if the data volume does not exceed the maximum data volume, linear fitting is performed on the historical data, so as to obtain a query time fitting curve representing a correspondence relationship between the query data volume and the query time.
When the storage space of the target data table in the expected filing time is insufficient, the filing time of the target data table can be adjusted, so that the adjusted filing time is before the expected filing time, the target data table can be stored in a distributed mode according to preset rules, the query route of the target data table is modified, and the problem that the storage space of the target data table in the expected filing time is insufficient is solved.
Based on the query timeout predicting method disclosed in the above embodiments, this embodiment correspondingly discloses a query timeout predicting device, please refer to fig. 4, which includes:
a data amount obtaining unit 401, configured to obtain a data amount of the target data table at the expected filing time;
a linear fitting unit 402, configured to perform linear fitting on the historical data to obtain a query time fitting curve representing a correspondence between a query data amount and query time;
a query time prediction unit 403, configured to determine, according to the query time fitting curve, an estimated query time corresponding to a data amount of the target data table at an estimated archiving time;
a query timeout determining unit 404, configured to determine whether there is a query timeout in the target data table at the expected archiving time based on the estimated query time.
Optionally, the data amount obtaining unit 401 includes:
the filing time acquiring subunit is used for acquiring the expected filing time of the target data table;
and the data quantity prediction subunit is used for predicting the data quantity of the target data table at the expected filing time according to the historical data.
Optionally, the filing time obtaining subunit is specifically configured to:
analyzing the filing rule of the target data table in historical data;
and determining the expected filing time of the target data table according to the filing rule of the target data table.
Optionally, the data amount predictor is specifically configured to:
performing linear fitting on the data quantity of the target data table at different time in the historical data to obtain a data quantity fitting curve representing the corresponding relation between the data quantity and the time;
and according to the data volume fitting curve, predicting the data volume of the target data table at the expected filing time.
Optionally, the data amount predictor is specifically configured to:
acquiring time characteristics of expected filing time;
inputting the time characteristics of the expected filing time into a pre-constructed data quantity prediction model for processing to obtain the data quantity of the target data table at the expected filing time, wherein the data quantity prediction model is obtained by training and verifying a neural network model by using a training sample marked with the data quantity in advance, and the training sample is the data quantity sample of the target data table under different time characteristics.
Optionally, the apparatus further comprises:
and the filing time adjusting unit is used for adjusting the filing time of the target data table to enable the adjusted filing time to be before the expected filing time when judging that the target data table has query timeout in the expected filing time.
Optionally, the apparatus further comprises:
and the storage optimization unit is used for performing distributed storage on the target data table according to a preset rule and modifying the query route of the target data table under the condition that the query timeout of the target data table in the expected filing time is judged.
Optionally, the apparatus further comprises:
the storage space judging unit is used for judging whether the data volume of the target data table in the expected filing time exceeds the maximum data volume; if the maximum data volume is exceeded, prompting that the storage space of the target data table in the expected filing time is insufficient; and if the maximum data volume is not exceeded, triggering the linear fitting unit.
The query timeout predicting device disclosed in this embodiment obtains a query time fitting curve representing a corresponding relationship between a query data amount and query time by performing linear fitting on historical data, and can accurately determine estimated query time corresponding to the data amount of a target data table at an estimated filing time according to the fitting curve, so as to determine whether query timeout exists in the estimated filing time of the target data table based on the estimated query time, thereby facilitating adjustment of the filing time or optimization of storage of the target data table as early as possible under the condition that query timeout exists in the estimated filing time of the target data table, avoiding occurrence of insufficient system storage space error reporting or query timeout error reporting due to an excessively large data amount of the target data table, and improving query efficiency.
It should be noted that the query timeout prediction method and apparatus provided by the present invention can be applied to the distributed field or the financial field. The above is merely an example, and the application fields of the query timeout prediction method and the query timeout prediction device provided by the present invention are not limited.
The embodiments in the present description are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.
It is further noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
The above embodiments can be combined arbitrarily, and the features described in the embodiments in the present specification can be replaced or combined with each other in the above description of the disclosed embodiments, so that those skilled in the art can implement or use the present application.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. A query timeout prediction method, comprising:
acquiring the data volume of the target data table in the expected filing time;
performing linear fitting on the historical data to obtain a query time fitting curve representing the corresponding relation between the query data amount and the query time;
according to the query time fitting curve, determining estimated query time corresponding to the data volume of the target data table at the estimated filing time;
and judging whether the target data table has query timeout at the expected filing time or not based on the estimated query time.
2. The method of claim 1, wherein obtaining the amount of data at the time of archiving the target data table comprises:
acquiring the expected filing time of the target data table;
and predicting the data amount of the target data table at the expected filing time according to the historical data.
3. The method of claim 2, wherein obtaining the expected archive time for the target data table comprises:
analyzing the filing rule of the target data table in historical data;
and determining the expected filing time of the target data table according to the filing rule of the target data table.
4. The method of claim 2, wherein predicting the amount of data for the target data table at the expected archive time based on historical data comprises:
performing linear fitting on the data quantity of the target data table at different time in the historical data to obtain a data quantity fitting curve representing the corresponding relation between the data quantity and the time;
and according to the data volume fitting curve, predicting the data volume of the target data table at the expected filing time.
5. The method of claim 2, wherein predicting the amount of data for the target data table at the expected archive time based on historical data comprises:
acquiring time characteristics of expected filing time;
inputting the time characteristics of the expected filing time into a pre-constructed data quantity prediction model for processing to obtain the data quantity of the target data table at the expected filing time, wherein the data quantity prediction model is obtained by training and verifying a neural network model by using a training sample marked with the data quantity in advance, and the training sample is the data quantity sample of the target data table under different time characteristics.
6. The method of claim 1, further comprising:
and when the target data table is judged to have query timeout in the expected archiving time, the archiving time of the target data table is adjusted, and the adjusted archiving time is made to be before the expected archiving time.
7. The method of claim 1, further comprising:
and under the condition that the query timeout of the target data table in the expected filing time is judged, performing distributed storage on the target data table according to a preset rule, and modifying the query route of the target data table.
8. The method of claim 1, wherein after obtaining the amount of data for the target data table at the expected archive time, the method further comprises:
judging whether the data volume of the target data table in the expected filing time exceeds the maximum data volume;
if the maximum data volume is exceeded, prompting that the storage space of the target data table in the expected filing time is insufficient;
and if the maximum data volume is not exceeded, performing linear fitting on the historical data to obtain a query time fitting curve representing the corresponding relation between the query data volume and the query time.
9. A query timeout predicting apparatus, comprising:
the data volume acquisition unit is used for acquiring the data volume of the target data table at the expected filing time;
the linear fitting unit is used for performing linear fitting on the historical data to obtain a query time fitting curve representing the corresponding relation between the query data amount and the query time;
the query time prediction unit is used for determining the estimated query time corresponding to the data volume of the target data table at the estimated filing time according to the query time fitting curve;
and the query timeout judging unit is used for judging whether the target data table has query timeout at the expected filing time or not based on the estimated query time.
10. The apparatus according to claim 9, wherein the data amount obtaining unit includes:
the filing time acquiring subunit is used for acquiring the expected filing time of the target data table;
and the data quantity prediction subunit is used for predicting the data quantity of the target data table at the expected filing time according to the historical data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111339471.1A CN114116835A (en) | 2021-11-12 | 2021-11-12 | Query timeout prediction method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111339471.1A CN114116835A (en) | 2021-11-12 | 2021-11-12 | Query timeout prediction method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114116835A true CN114116835A (en) | 2022-03-01 |
Family
ID=80379120
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111339471.1A Pending CN114116835A (en) | 2021-11-12 | 2021-11-12 | Query timeout prediction method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114116835A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7127456B1 (en) * | 2002-12-05 | 2006-10-24 | Ncr Corp. | System and method for logging database queries |
| WO2016033973A1 (en) * | 2014-09-05 | 2016-03-10 | 中兴通讯股份有限公司 | Method and system for predicting resource occupancy |
| CN110096489A (en) * | 2019-04-30 | 2019-08-06 | 阿里巴巴集团控股有限公司 | A kind of data query method, system, device and electronic equipment |
| CN111782872A (en) * | 2020-06-30 | 2020-10-16 | 北京百度网讯科技有限公司 | Query method, query device, electronic equipment and storage medium |
| CN113360357A (en) * | 2021-06-01 | 2021-09-07 | 深圳前海微众银行股份有限公司 | Data monitoring method, system, equipment and computer program product |
-
2021
- 2021-11-12 CN CN202111339471.1A patent/CN114116835A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7127456B1 (en) * | 2002-12-05 | 2006-10-24 | Ncr Corp. | System and method for logging database queries |
| WO2016033973A1 (en) * | 2014-09-05 | 2016-03-10 | 中兴通讯股份有限公司 | Method and system for predicting resource occupancy |
| CN110096489A (en) * | 2019-04-30 | 2019-08-06 | 阿里巴巴集团控股有限公司 | A kind of data query method, system, device and electronic equipment |
| CN111782872A (en) * | 2020-06-30 | 2020-10-16 | 北京百度网讯科技有限公司 | Query method, query device, electronic equipment and storage medium |
| CN113360357A (en) * | 2021-06-01 | 2021-09-07 | 深圳前海微众银行股份有限公司 | Data monitoring method, system, equipment and computer program product |
Non-Patent Citations (1)
| Title |
|---|
| 武珊珊;谷峪;岳德君;于戈;: "一种数据流上基于截止期的多查询过载预测模型", 东北大学学报(自然科学版), no. 07, 15 July 2007 (2007-07-15), pages 961 - 965 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110417591B (en) | Voting node configuration method and system | |
| CN105451260B (en) | Network request method, network fluctuation measuring method and device | |
| US20140325151A1 (en) | Method and system for dynamically managing big data in hierarchical cloud storage classes to improve data storing and processing cost efficiency | |
| CN101635651A (en) | Method, system and device for managing network log data | |
| US8341007B2 (en) | Systems and methods for forecasting demand of an object in a managed supply chain | |
| US11494413B1 (en) | Query alerts generation for virtual warehouse | |
| CN111010303A (en) | Server control method and device | |
| CN109413694B (en) | Small cell caching method and device based on content popularity prediction | |
| CN110737717B (en) | Database migration method and device | |
| CN114528231A (en) | Data dynamic storage method and device, electronic equipment and storage medium | |
| CN114866563A (en) | Capacity expansion method, device, system and storage medium | |
| CN117666947A (en) | Data storage method, device, electronic equipment and computer readable medium | |
| CN117950599B (en) | I/O stack construction method, device, equipment and medium based on distributed system | |
| US20210149868A1 (en) | Dynamic index management for computing storage resources | |
| CN114116835A (en) | Query timeout prediction method and device | |
| US20230334052A1 (en) | Dynamic Query Allocation to Virtual Warehouses | |
| JP4526774B2 (en) | Apparatus and method for performing performance balance evaluation and sizing of computer system components | |
| CN116128132A (en) | Power consumption prediction method, device, equipment and medium | |
| CN113411224B (en) | Data processing method and device, electronic equipment and storage medium | |
| US11914595B2 (en) | Virtual warehouse query monitoring and reporting | |
| WO2022062777A1 (en) | Data management method, data management apparatus, and storage medium | |
| CN107908555A (en) | A kind of method for detecting abnormality and its terminal of SQL scripts | |
| CN115344606A (en) | Data query method, system, server and storage medium based on secondary cache | |
| CN114020218A (en) | Mixed repeating data deleting and scheduling method and system | |
| CN114143263A (en) | Method, device and medium for limiting current of user request |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |