CN117453489A - Detection method, detection device, electronic equipment and storage medium - Google Patents

Detection method, detection device, electronic equipment and storage medium Download PDF

Info

Publication number
CN117453489A
CN117453489A CN202311617763.6A CN202311617763A CN117453489A CN 117453489 A CN117453489 A CN 117453489A CN 202311617763 A CN202311617763 A CN 202311617763A CN 117453489 A CN117453489 A CN 117453489A
Authority
CN
China
Prior art keywords
data
time series
series data
target
historical
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202311617763.6A
Other languages
Chinese (zh)
Inventor
徐征
李清颢
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Agricultural Bank of China
Original Assignee
Agricultural Bank of China
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Agricultural Bank of China filed Critical Agricultural Bank of China
Priority to CN202311617763.6A priority Critical patent/CN117453489A/en
Publication of CN117453489A publication Critical patent/CN117453489A/en
Pending legal-status Critical Current

Links

Landscapes

  • Debugging And Monitoring (AREA)

Abstract

The invention discloses a detection method, a detection device, electronic equipment and a storage medium. The detection method comprises the following steps: acquiring at least one historical time series data, wherein the historical time series data comprise time-ordered usage of data discs corresponding to a distributed relational database; preprocessing the historical time series data to obtain processed historical time series data; inputting the processed historical time series data into a processing model, and predicting target time series data of target time; and detecting the use condition of the data disc based on the target time sequence data. The technical problem that the use condition of the data disk is abnormal and needs to be manually analyzed and found is solved, and the use condition of the data disk is reflected by the data volume of the data disk, so that the data disk is monitored. The method has the advantages that the target time sequence data of the target time is estimated through the processing model, the service condition of the data disc is estimated in advance, and the early warning effect is achieved.

Description

Detection method, detection device, electronic equipment and storage medium
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a detection method, a detection device, an electronic device, and a storage medium.
Background
The monitoring of the business index data of daily operation and maintenance is basically based on the experience of operation and maintenance personnel. When the data increment synchronization from the distributed relational database to other databases is carried out, if uncommon transactions exist in the distributed relational database, the use condition of the data disk such as the fact that log file accumulation cannot be automatically cleaned is caused.
The data and the logs in the distributed relational database are placed on the same disk, so that abnormal use conditions of the data disk, such as incapability of cleaning up log file accumulation, can be found only through active query analysis of operation and maintenance personnel, and if the problems occur at night or in busy hours, more serious problems can be caused.
Disclosure of Invention
The invention provides a detection method, a detection device, electronic equipment and a storage medium, which are used for solving the technical problem that the use condition of a data disc is abnormal and needs to be manually analyzed and found.
According to an aspect of the present invention, there is provided a detection method including:
acquiring at least one historical time series data, wherein the historical time series data comprise time-ordered usage of data discs corresponding to a distributed relational database;
preprocessing the historical time series data to obtain processed historical time series data;
Inputting the processed historical time series data into a processing model, and predicting target time series data of target time;
and detecting the use condition of the data disc based on the target time sequence data.
According to another aspect of the present invention, there is provided a detection apparatus including:
the acquisition module is used for acquiring at least one historical time series data, wherein the historical time series data comprises time-ordered usage amounts of data discs corresponding to the distributed relational database;
the preprocessing module is used for preprocessing the historical time series data to obtain processed historical time series data;
the estimating module is used for inputting the processed historical time series data into the processing model and estimating target time series data of target time;
and the detection module is used for detecting the use condition of the data disk based on the target time sequence data.
According to another aspect of the present invention, there is provided an electronic apparatus including:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,
the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the detection method according to any one of the embodiments of the present invention.
According to another aspect of the present invention, there is provided a computer readable storage medium storing computer instructions for causing a processor to execute the detection method according to any one of the embodiments of the present invention.
According to the technical scheme, at least one historical time series data is obtained, wherein the historical time series data comprise time-ordered usage amounts of data discs corresponding to a distributed relational database; preprocessing the historical time series data to obtain processed historical time series data; inputting the processed historical time series data into a processing model, and predicting target time series data of target time; the use condition of the data disk is detected based on the target time sequence data, the technical problem that the use condition of the data disk is abnormal and needs to be manually analyzed and found is solved, and the use condition of the data disk is reflected by the data volume of the data disk, so that the data disk is monitored. The method has the advantages that the target time sequence data of the target time is estimated through the processing model, the service condition of the data disc is estimated in advance, and the early warning effect is achieved.
It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the invention or to delineate the scope of the invention. Other features of the present invention will become apparent from the description that follows.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a detection method according to a first embodiment of the present invention;
FIG. 2 is a flow chart of a detection method according to a second embodiment of the present invention;
FIG. 3 is a schematic diagram of a data disk usage prediction effect according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a detection device according to a third embodiment of the present invention;
fig. 5 is a schematic structural diagram of an electronic device implementing a detection method according to an embodiment of the present invention.
Detailed Description
In order that those skilled in the art will better understand the present invention, a technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and the claims of the present invention and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the invention described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
The existing operation and maintenance monitoring system is visualized through a time sequence of statistical historical data, so that operation and maintenance personnel can clearly see fluctuation of indexes to analyze results, and when the displayed numerical value reaches or exceeds a specified value, the monitored system is considered to be abnormal. However, the current method only displays the historical data, and neither the regular mining of the data nor the analysis of the future trends is involved.
According to the invention, the previous monitoring indexes of the distributed relational database are analyzed and learned through a deep learning method, continuous learning can be carried out more specifically, the service habit change of the GaussDB database or the change caused by hardware configuration or operating system level change is adapted through continuously rolling and updating the training data set, and the pre-warning can be carried out on the anomaly which happens silently while the continuous learning is carried out.
Example 1
Fig. 1 is a flowchart of a detection method according to a first embodiment of the present invention, where the method may be applied to detecting a usage situation of a data disc, and the method may be performed by a detection device, where the detection device may be implemented in a form of hardware and/or software, and the detection device may be configured in an electronic device. The electronic device may be a computer, a cell phone, a personal digital assistant, and/or a server.
As shown in fig. 1, the method includes:
s110, acquiring at least one historical time series data.
The historical time series data includes time ordered usage of data disks corresponding to the distributed relational database. The distributed relational database can be an enterprise-level distributed relational database and can have key capabilities of high availability on the cloud, high reliability, high safety, elastic expansion, one-key deployment, quick backup and recovery, monitoring and alarming and the like. The data disk may be a disk on which data and logs of a distributed relational database are stored. The usage amount may be an amount of data disc storage space. The historical time series data may be data obtained by sorting the usage amounts by time. The historical time series data may be a collection of usage amounts of the data disk history.
The present operation may obtain at least one historical time series data locally or remotely. The time series data may be considered as a time ordered usage of the data disk corresponding to the distributed relational database. The time series data may be collected by a monitoring system. The monitoring system may be an alarm management platform such as Prometheus. The monitoring system is a service monitoring system and time sequence database, which provides a universal data model and a quick data acquisition, storage and query interface. The core component periodically pulls data from a statically configured monitoring target or a target automatically configured based on service discovery, and when the newly pulled data is larger than the configured memory cache area, the data is persisted into the storage device.
The present operation may obtain at least one historical time series data from the monitoring system. The number of historical time series data is not limited and may be associated with the number required to process model inputs.
S120, preprocessing the historical time series data to obtain processed historical time series data.
After the historical time series data is obtained, the operation can preprocess the historical time series data so as to facilitate inputting the processed historical time series data into the processing model.
The preprocessing means is not limited herein, and one or more of missing value filling, outlier removal, and noise cancellation may be performed on the historical time series data.
After one or more of the missing value padding, outlier removal, noise cancellation are performed on the historical time series data, normalization processing may be performed to convert to a value in the range of 0-1.
S130, inputting the processed historical time series data into a processing model, and estimating target time series data of target time.
The processing model may be considered as a model that implements time series data prediction. The process model may be a neural network model, such as a recurrent neural network model. The target time series data may be regarded as predicted time series data of the target time, characterizing the usage of the target time data disc.
The present operation inputs at least one processed historical time series data to the processing model, outputting the target time series data. The target time may be a time at some point in the future. If the target time is the next time of the latest time in a plurality of times corresponding to the at least one historical time series data. The time granularity is not limited here.
And S140, detecting the use condition of the data disk based on the target time sequence data.
After the target time series data is obtained, the operation can analyze the target time series data to realize the estimation of the service condition of the data disk.
The operation can display the target time series data and analyze the use condition of the data disc in a visual mode. In visualizing the target time-series data, the target time-series data may be plotted in the form of a curve.
The operation can also compare the threshold value of the usage amount corresponding to the target time series data, and determine the usage condition of the data disc when the target time corresponding to the target time series data is determined.
The embodiment provides a detection method, wherein data to be backed up is backed up into a storage container on the cloud through a creation backup interface of a distributed relational database; determining an instance to be backed up in the storage container on the cloud through a backup list interface of the distributed relational database, wherein the instance to be backed up is an instance to be backed up in the backup data; backing up the instance to be backed up from the cloud storage container to a data backup system; the data backup system is used for backing up the instance to be backed up to the target data center, so that the technical problem of limitation of a backup mode of the distributed relational database is solved, the backup instance in the distributed relational data is backed up to any target data center through the on-cloud storage center and the data backup system, and the target data center to which the backup is carried out is free of limitation of a platform and a region.
On the basis of the above embodiments, modified embodiments of the above embodiments are proposed, and it is to be noted here that only the differences from the above embodiments are described in the modified embodiments for the sake of brevity of description.
In one embodiment, the acquiring at least one historical time series data includes:
acquiring historical monitoring data of the data disk from a monitoring system, wherein the historical monitoring data comprises time-ordered usage of the data disk;
and acquiring at least one historical time series data with a first set length from the historical monitoring data.
The history monitoring data may be regarded as monitoring data of the history of the data disc. The monitoring data may be monitoring of usage of the data disc. The invention obtains, stores, uses, processes and the like the data all conform to the relevant regulations of the relevant laws.
The first set length may be a sequence length of historical time series data set in advance.
In this embodiment, the electronic device may acquire the history monitoring data of the data disc from the monitoring system, and then extract a plurality of historical time series data of the first set length from the history monitoring data. For example, the historical time series data of the first set length is extracted from the historical monitoring data every time according to time until the number of the extracted historical time series data meets the requirement of the required number.
In one embodiment, the preprocessing the historical time series data to obtain processed historical time series data includes:
and carrying out missing value filling, noise removal and normalization processing on the historical time series data to obtain the processed historical time series data.
The missing value padding may be considered to be padding of missing values in the historical time series data, and the specific means is not limited. The preprocessing may include data cleansing of the historical time series data. Noise may be interference data in the historical time series data, i.e. random errors or variances in the historical time series data. Noise may be removed when the data is cleaned.
In the embodiment, the normalization processing is performed on the historical time series data, so that the data is limited in a certain range, and the processing model processing is facilitated.
The order of the missing value filling, noise removal, and normalization is not limited. The normalization processing operation may be performed last.
In one embodiment, the target time is a next time point of a latest time in the times corresponding to the at least one historical time series data.
The embodiment can analyze data according to the set time granularity. Such as data analysis at time granularity every n minutes. Each time may correspond to a time series of data, e.g., the time granularity is 10 minutes, and the nine points correspond to a time series of data, indicating nine to nine ten minutes of usage. The nine points correspond to one time series data, indicating the usage amount of nine points to twenty points.
Each historical time series data corresponds to a time point, the latest time is selected from the times corresponding to all the historical time series data, and the next time point of the latest time is taken as the target time. If the latest time is nine, the next time is nine.
In one embodiment, the detecting the usage of the data disc based on the target time-series data includes:
displaying a curve corresponding to the target time sequence data in a target interface;
and after the target time arrives, displaying the actual time sequence data corresponding to the target time into the target interface.
The target interface may be considered as an interface for displaying a curve corresponding to time-series data, such as displaying a curve corresponding to target time-series data and a curve corresponding to actual time-series data.
The embodiment can display the curve corresponding to the target time series data through the target interface so as to show the target time series data in a visual mode.
The actual time-series data may be regarded as time-series data corresponding to the actual usage amount of the data disc after the target time has arrived.
The embodiment displays the actual time series data to the target interface after the target time arrives.
In the embodiment, a curve corresponding to the target time sequence data and a curve corresponding to the actual time sequence data can be displayed in the target interface at the same time, so that the estimated target time sequence data and the actual time sequence data are compared, and the estimated result is checked, so that the processing model is convenient to adjust.
In one embodiment, the detecting the usage of the data disc based on the target time-series data includes:
determining a target usage amount corresponding to the target time sequence data;
and under the condition that the target usage amount is larger than a preset threshold value, determining that the usage condition of the data disc corresponding to the target time is abnormal.
The target usage amount may be regarded as a usage amount of the data disc corresponding to the target time series data.
In this embodiment, when determining the target usage amount, the usage amount included in the target time series data may be analyzed, and the analysis means is not limited, for example, the average value may be directly obtained, or the average value may be obtained after removing the abnormal data. Or perform the rest of the mathematical operations.
After determining the target usage, the target usage may be compared to a set threshold. In the case where the target usage amount is greater than the set threshold, it can be considered that there will be an abnormality in the data disk at the target time.
In an embodiment, the detection method further comprises:
updating a training set according to a set period, wherein the set period is associated with a service period of the distributed relational database;
and updating the processing model based on the training set to obtain an updated post-processing model.
The training set may be a sample set of training process models. The set period may be a preset period of updating the training set. The traffic cycle may be a processing cycle of the traffic. The set period may be equal to the traffic period. After the service period arrives, the set period arrives, thereby triggering updating the training set. And if the time sequence data which is not included in the actually generated training set is added into the training set, the training set is periodically updated.
After the training set is updated, the processing model can be updated based on the updated training set, and the updated processing model is obtained. Such as adjusting model parameters of the process model by the updated training set.
Example two
The present embodiment describes a detection method by way of example. The detection method in the embodiment can be regarded as a detection and prediction method for automatic cleaning failure of the xlog log of the distributed relational database based on deep learning. xlog, i.e., write Ahead Log (WAL), redox Log. If the data file is to be modified, the modification must be made after these modification operations have been logged to the log file, i.e. after the log record describing these changes has been flushed to persistent storage.
The business index data of daily operation and maintenance can have some obvious abnormality of yesterday, obvious problem of continuous deviation, index data drifting along with time period and the like. Previously, the configuration of the monitoring is basically based on engineer experience or continuous iterative correction, even purely by manual investigation, but in the face of a large amount of monitoring data, it may be difficult for an operation and maintenance person to find a problem or to directly find a potential alarm point. On the other hand, the vast amount of monitoring data generated by monitoring systems does not fully play its value, and these data can be used to discover and mine out certain connections that may be useful therein. And for complex database systems, a single historical data monitoring curve is difficult to analyze and locate problems in a short period of time. With the development of monitoring systems, operation and maintenance standardization and automation can be achieved by formulating monitoring standards and automated monitoring deployment, and the final goal is to thoroughly solve the problem by an intelligent method.
When the data increment synchronization from the distributed relational database, such as a Gaussian database, to other databases is performed through the data replication service (Data Replication Service, DRS), if an uncommitted transaction exists at the end of the distributed relational database, the DRS logical replication slot is caused to stop advancing, so that xlog accumulation cannot be automatically cleaned. Wherein the DRS is a cloud service for database live migration and database live synchronization.
The same problem can occur when the distributed relational database performs backup recovery tasks, if a backup failure occurs. Because the database data and logs of the distributed relational database need to be placed on the same disk, the discovery of such serious problems can only be analyzed by an administrator through active query, and if such problems occur at night or during busy hours, the problems can be more serious and the problems are more difficult to locate to the root cause. Such monitoring indicators cannot simply alert of anomalies by a simple fixed threshold, but rather are discovered and processed at the beginning of the situation occurrence, avoiding subsequent serious problems.
The method uses a gating circulation unit (Gate Recurrent Unit, GRU) to predict the database monitoring curve, and comprises the following steps:
(1) Collecting and preparing data: monitoring curve data first needs to be collected from a database and prepared into a format that can be used for training and testing models. In general, the monitoring data for each time point can be taken as one input sample, and then the sample length (e.g., 5 minutes, 10 minutes, etc.) and the prediction target (e.g., the value of the next time point) can be set as needed.
(2) And (3) constructing a model: and constructing a proper GRU model according to the characteristics and the predicted targets of the data. In general, the model should contain at least one GRU layer, and possibly a fully connected or convolutional layer, etc. In addition, it is also necessary to select an appropriate loss function and optimization algorithm.
(3) Training a model: model training is performed using the prepared data. Different hyper-parameters and model structures can be tried in the training process so as to achieve better effects.
(4) Test model: the test set data is used to evaluate the performance of the model that has been trained. The evaluation results can be used to optimize the hyper-parameters and structure of the model.
(5) Predicting future values: the monitoring curve at future time points is predicted using the already trained model. In general, the value of the next point in time can be predicted using the monitoring data of the previous points in time as input, then added to the monitoring curve, and the prediction of the value of the next point in time continued, and so on.
(6) Continuous learning: changes to business changes or other operating system level changes are made by rolling updates to the training data set, i.e., updating the training set at set periods.
The method for predicting the database monitoring curve by using the GRU comprises the following specific steps:
(1) Data acquisition and preprocessing
In order to monitor the usage of the data disk of the distributed relational database, the related monitoring data needs to be collected from the distributed relational database system. The present invention selects the use of the monitoring system to collect time series data (i.e., obtain at least one historical time series data) in a distributed environment. The data collected by the monitoring system can be directly imported into the monitoring instrument system for display. By setting proper monitoring indexes, the monitoring of the usage amount of the data disc of the distributed relational database can be effectively realized. The monitoring data obtained from the monitoring system may have problems such as missing values, abnormal values, noise, and the like. Therefore, these data need to be preprocessed before processing using the GRU model (i.e., the processing model). The pretreatment step comprises the following steps: filling the missing value, removing the abnormal value, and eliminating noise. And preprocessing the historical time series data to obtain processed historical time series data.
(2) Model design
The invention selects GRU as the processing model of the monitoring data to realize data anomaly detection and future prediction. The GRU is a recurrent neural network model, can process time series data and has certain memory capacity. In the present invention, the input layer uses the GRU and then outputs as the full connection layer. Each node of the input layer contains a gate structure for controlling the flow of information and each node contains a reset gate and an update gate. The reset gate is used to control the interaction of the newly acquired input signal with the history, while the update gate is used to determine the ratio of the new state to the current state. In the GRU model, it is more efficient than the LSTM model due to the control of the gating mechanism. The following is a step of model design using the GRU: 1) Defining inputs and outputs-first, the input x and the output y need to be defined. A sequence with a length of T is set, wherein each element is d-dimension, and the sequence length of input x and output y can be the same or different. 2) Defining parameters next, parameters to be defined include Wz, wr, wh, uz, ur, uh and bz, br, bh. Where Wz, wr, wh, uz, ur and Uh are weight matrices and bz, br, and bh are bias vectors. These parameters will be learned during the training process. 3) Defining a gating unit, wherein the GRU comprises two gating units: reset gates and update gates. Resetting the gate helps to determine the effect of the state of the previous time step on the current state, and updating the gate helps to determine the information to be propagated from the previous state to the current state. These gates are controlled by a sigmoid function. 4) Definition memory unit the GRU also contains a memory unit c which stores the information in the sequence and passes it on to the next time step. 5) And calculating output, namely finally, calculating output y. This is done by multiplying the hidden state of the GRU with a weight matrix using a softmax function.
(3) Model training
In the training process, all acquired data are divided into a training set and a testing set. For anomaly detection, the invention adopts epsilon elimination method, namely, a proper threshold is selected as a judgment standard, and the data exceeding the threshold is regarded as anomaly data (namely, the anomaly exists in the use condition of the data disc corresponding to the target time under the condition that the target use amount is larger than a preset threshold). In the model training process, the Mean Square Error (MSE) is used as a loss function for optimization, and Adam algorithm is used for parameter optimization. After training is completed, the model is exported for use in subsequent data prediction.
(4) Data prediction
Future data predictions are made on the trained model. For example, the disk usage for the next 5 days can be predicted from the data of the first 10 days (i.e., the first set length). The prediction results can help an administrator to discover potential problems in time to take corresponding measures, such as increasing storage space or adjusting storage strategies. Before data prediction, normalization processing is required for the acquired data. Since there may be differences in the data value sizes at different points in time, it is necessary to uniformly convert them into values in the range of 0 to 1. The invention adopts a standard deviation normalization method for processing. Then, a trained GRU model is input, and the disk usage for the next 5 days is predicted from the data of the previous 10 days. The predicted results may be presented to an administrator for review in the form of charts and reports.
(5) Continuous learning
Compared with one-time training, the method provided by the invention has the advantages that the training set can be continuously updated for continuous learning, so that the model is continuously updated to follow the change, the updating of the training set can be adjusted in days, weeks and months according to the service period of the database (namely, the training set is updated according to the set period, the set period is associated with the service period of the distributed relational database), and the old training set is updated by using the periodically generated new data.
Fig. 2 is a flowchart of yet another detection method according to the second embodiment of the present invention, referring to fig. 2, in this embodiment, the production and storage of data and logs are performed by a distributed relational database production server, i.e. a server corresponding to the distributed relational database. The monitoring system server can monitor time sequence data. After the historical time series data is acquired by the monitoring system, data preprocessing operations may be performed to facilitate training and updating of the model (i.e., the process model). When the processing model is trained or updated, whether the accuracy of the processing model meets the standard can be determined, if not, the processing model is continuously trained or updated, if yes, whether the set period threshold is reached, namely whether the set period of the updated training set is reached, and if yes, the processing model is updated after the training set is updated. If the period threshold is not reached, the model can be transmitted to a prediction server, and the prediction of the target time series data and the detection of the abnormality can be realized through the prediction server. The prediction server may be a server that implements the target time series data prediction.
Fig. 3 is a schematic diagram of a data disk usage prediction effect according to an embodiment of the present invention, where the data disk usage is the usage of a data disk. FIG. 3 is a schematic diagram showing the target interface displaying the target time series data and the actual time series data, with the solid line being the target time series data of the target time estimated by the process model. The dashed line is the actual time series data when the target time arrives. As can be seen from fig. 3, the estimated result of the process model deviates less from the actual usage amount.
The invention analyzes the high-dimensional performance index by using a deep learning model (i.e. a processing model) and extracts hidden characteristics, thereby realizing real-time monitoring and anomaly detection. In the model training process, newly generated monitoring data are added into a continuous training set, and the model is continuously updated and optimized through incremental learning, online learning and other modes. The invention can predict future data, the prediction result given by the continuously learned model can be more practical and the influence caused by operating system change and database change can be eliminated. The invention can also be combined with a visual tool to display the analysis result so that the operation and maintenance personnel can better understand and process the analysis result.
Example III
Fig. 4 is a schematic structural diagram of a detection device according to a third embodiment of the present invention. As shown in fig. 3, the apparatus includes:
an obtaining module 410, configured to obtain at least one historical time series data, where the historical time series data includes time-ordered usage amounts of data disks corresponding to the distributed relational database;
a preprocessing module 420, configured to preprocess the historical time series data to obtain processed historical time series data;
the estimating module 430 is configured to input the processed historical time series data into the processing model, and estimate target time series data of the target time;
and a detection module 440, configured to detect a usage situation of the data disc based on the target time-series data.
In one embodiment, the obtaining module 410 is specifically configured to:
acquiring historical monitoring data of the data disk from a monitoring system, wherein the historical monitoring data comprises time-ordered usage of the data disk;
and acquiring at least one historical time series data with a first set length from the historical monitoring data.
In one embodiment, the preprocessing module 420 is specifically configured to:
And carrying out missing value filling, noise removal and normalization processing on the historical time series data to obtain the processed historical time series data.
In one embodiment, the target time is a next time point of a latest time in the times corresponding to the at least one historical time series data.
In one embodiment, the detection module 440 is specifically configured to:
displaying a curve corresponding to the target time sequence data in a target interface;
and after the target time arrives, displaying the actual time sequence data corresponding to the target time into the target interface.
In one embodiment, the detection module 440 is specifically configured to:
determining a target usage amount corresponding to the target time sequence data;
and under the condition that the target usage amount is larger than a preset threshold value, determining that the usage condition of the data disc corresponding to the target time is abnormal.
In one embodiment, the detecting device further includes an updating module configured to:
updating a training set according to a set period, wherein the set period is associated with a service period of the distributed relational database;
and updating the processing model based on the training set to obtain an updated post-processing model.
The detection device provided by the embodiment of the invention can execute the detection method provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.
Example IV
Fig. 5 is a schematic structural diagram of an electronic device implementing a detection method according to an embodiment of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device 10 may also represent various forms of mobile equipment, such as personal digital assistants, cellular telephones, smartphones, wearable devices (e.g., helmets, eyeglasses, watches, etc.), and other similar computing equipment. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.
As shown in fig. 5, the electronic device 10 includes at least one processor 11, and a Memory, such as a Read-Only Memory (ROM) 12, a random access Memory (Random Access Memory, RAM) 13, etc., communicatively connected to the at least one processor 11, wherein the Memory stores a computer program executable by the at least one processor 11, the computer program being executed by the at least one processor 11 to enable the at least one processor 11 to perform the method provided by the present invention.
The processor 11 may perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM) 12 or a computer program loaded from a storage unit 18 into a Random Access Memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 may also be stored. The processor 11, the ROM 12 and the RAM 13 are connected to each other via a bus 14. An input/output (I/O) interface 15 is also connected to bus 14.
Various components in the electronic device 10 are connected to the I/O interface 15, including: an input unit 16 such as a keyboard, a mouse, etc.; an output unit 17 such as various types of displays, speakers, and the like; a storage unit 18 such as a magnetic disk, an optical disk, or the like; and a communication unit 19 such as a network card, modem, wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.
The processor 11 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU), various dedicated artificial intelligence (Artificial Intelligence, AI) computing chips, various processors running machine learning model algorithms, digital signal processors (Digital Signal Process, DSP), and any suitable processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as the detection method.
In some embodiments, the detection method may be implemented as a computer program tangibly embodied on a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 10 via the ROM 12 and/or the communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the detection method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the detection method in any other suitable way (e.g. by means of firmware).
Various implementations of the systems and techniques described here above can be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (Field Programmable Gate Array, FPGAs), application specific integrated circuits (Application Specific Integrated Circuit, ASICs), application specific standard products (Application Specific Standard Parts, ASSPs), systems On Chip (SOC), complex programmable logic devices (Complex Programmable logic device, CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.
A computer program for carrying out methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be implemented. The computer program may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of the present invention, a computer-readable storage medium stores computer instructions for causing a processor to execute the detection method provided by the present invention.
A computer readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-Only Memory (ROM), an erasable programmable read-Only Memory (Erasable Programmable Read-Only Memory, EPROM or flash Memory), an optical fiber, a compact disc read-Only Memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having: other types of devices may also be used to provide interaction with the user, for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user may be received in any form (including acoustic input, speech input, or tactile input).
The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local area network (Local Area Network, LAN), wide area network (Wide Area Network, WAN), blockchain network, and the internet.
The computing system may include clients and servers. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical host and virtual special server (Virtual Private Server, VPS) service are overcome.
It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps described in the present invention may be performed in parallel, sequentially, or in a different order, so long as the desired results of the technical solution of the present invention are achieved, and the present invention is not limited herein.
The above embodiments do not limit the scope of the present invention. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of the present invention.

Claims (10)

1. A method of detection comprising:
acquiring at least one historical time series data, wherein the historical time series data comprise time-ordered usage of data discs corresponding to a distributed relational database;
preprocessing the historical time series data to obtain processed historical time series data;
inputting the processed historical time series data into a processing model, and predicting target time series data of target time;
and detecting the use condition of the data disc based on the target time sequence data.
2. The method of claim 1, wherein the acquiring at least one historical time series data comprises:
acquiring historical monitoring data of the data disk from a monitoring system, wherein the historical monitoring data comprises time-ordered usage of the data disk;
and acquiring at least one historical time series data with a first set length from the historical monitoring data.
3. The method of claim 1, wherein preprocessing the historical time series data to obtain processed historical time series data comprises:
And carrying out missing value filling, noise removal and normalization processing on the historical time series data to obtain the processed historical time series data.
4. The method of claim 1, wherein the target time is a next time point of a latest time among times corresponding to the at least one historical time series data.
5. The method of claim 1, wherein the detecting the usage of the data disc based on the target time-series data comprises:
displaying a curve corresponding to the target time sequence data in a target interface;
and after the target time arrives, displaying the actual time sequence data corresponding to the target time into the target interface.
6. The method of claim 1, wherein the detecting the usage of the data disc based on the target time-series data comprises:
determining a target usage amount corresponding to the target time sequence data;
and under the condition that the target usage amount is larger than a preset threshold value, determining that the usage condition of the data disc corresponding to the target time is abnormal.
7. The method as recited in claim 1, further comprising:
Updating a training set according to a set period, wherein the set period is associated with a service period of the distributed relational database;
and updating the processing model based on the training set to obtain an updated post-processing model.
8. A detection apparatus, characterized by comprising:
the acquisition module is used for acquiring at least one historical time series data, wherein the historical time series data comprises time-ordered usage amounts of data discs corresponding to the distributed relational database;
the preprocessing module is used for preprocessing the historical time series data to obtain processed historical time series data;
the estimating module is used for inputting the processed historical time series data into the processing model and estimating target time series data of target time;
and the detection module is used for detecting the use condition of the data disk based on the target time sequence data.
9. An electronic device, the electronic device comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,
the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the detection method of any one of claims 1-7.
10. A computer readable storage medium, characterized in that the computer readable storage medium stores computer instructions for causing a processor to implement the detection method of any one of claims 1-7 when executed.
CN202311617763.6A 2023-11-29 2023-11-29 Detection method, detection device, electronic equipment and storage medium Pending CN117453489A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311617763.6A CN117453489A (en) 2023-11-29 2023-11-29 Detection method, detection device, electronic equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311617763.6A CN117453489A (en) 2023-11-29 2023-11-29 Detection method, detection device, electronic equipment and storage medium

Publications (1)

Publication Number Publication Date
CN117453489A true CN117453489A (en) 2024-01-26

Family

ID=89596863

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311617763.6A Pending CN117453489A (en) 2023-11-29 2023-11-29 Detection method, detection device, electronic equipment and storage medium

Country Status (1)

Country Link
CN (1) CN117453489A (en)

Similar Documents

Publication Publication Date Title
US11403164B2 (en) Method and device for determining a performance indicator value for predicting anomalies in a computing infrastructure from values of performance indicators
CN111045894B (en) Database abnormality detection method, database abnormality detection device, computer device and storage medium
KR102440335B1 (en) A method and apparatus for detecting and managing a fault
CN111563524A (en) Multi-station fusion system operation situation abnormity monitoring and alarm combining method
CN116049146B (en) Database fault processing method, device, equipment and storage medium
JP7145821B2 (en) Failure probability evaluation system and method
CN111382494A (en) System and method for detecting anomalies in sensory data of industrial machines
CN115454778A (en) Intelligent monitoring system for abnormal time sequence indexes in large-scale cloud network environment
CN115858303B (en) Zabbix-based server performance monitoring method and system
CN113196311A (en) System and method for identifying and predicting abnormal sensing behavior patterns of a machine
US11928565B2 (en) Automated model building and updating environment
CN117194919A (en) Production data analysis system
KR101960755B1 (en) Method and apparatus of generating unacquired power data
JP7062505B2 (en) Equipment management support system
CN116862109A (en) Regional carbon emission situation awareness early warning method
CN117215258A (en) Numerical control machine tool real-time state monitoring system and method based on Flink
CN116203352A (en) Fault early warning method, device, equipment and medium for power distribution network
CN117453489A (en) Detection method, detection device, electronic equipment and storage medium
CN115118580B (en) Alarm analysis method and device
CN107121616B (en) Method and device for fault positioning of intelligent instrument
CN114881112A (en) System anomaly detection method, device, equipment and medium
CN111221479A (en) Method, system and storage medium for judging abnormal storage capacity variation
CN115829160B (en) Time sequence abnormality prediction method, device, equipment and storage medium
CN117514649B (en) Method and device for monitoring health state of wind turbine generator
CN117932437A (en) Method, apparatus and storage medium for predicting equipment failure

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination