CN117473268A - Threshold prediction method, system, equipment and storage medium - Google Patents
Threshold prediction method, system, equipment and storage medium Download PDFInfo
- Publication number
- CN117473268A CN117473268A CN202311501716.5A CN202311501716A CN117473268A CN 117473268 A CN117473268 A CN 117473268A CN 202311501716 A CN202311501716 A CN 202311501716A CN 117473268 A CN117473268 A CN 117473268A
- Authority
- CN
- China
- Prior art keywords
- prediction
- statistical information
- preset
- model
- threshold
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 72
- 238000003860 storage Methods 0.000 title claims abstract description 14
- 238000012549 training Methods 0.000 claims abstract description 43
- 230000008569 process Effects 0.000 claims abstract description 27
- 238000010801 machine learning Methods 0.000 claims abstract description 25
- 238000002360 preparation method Methods 0.000 claims description 21
- 238000013500 data storage Methods 0.000 claims description 11
- 238000012360 testing method Methods 0.000 claims description 11
- 238000012545 processing Methods 0.000 claims description 10
- 238000011156 evaluation Methods 0.000 claims description 7
- 238000009826 distribution Methods 0.000 claims description 6
- 238000000605 extraction Methods 0.000 claims description 6
- 238000004590 computer program Methods 0.000 claims description 4
- 238000007781 pre-processing Methods 0.000 claims description 4
- 238000012795 verification Methods 0.000 description 7
- 230000002159 abnormal effect Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 238000001514 detection method Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000005457 optimization Methods 0.000 description 3
- 230000005856 abnormality Effects 0.000 description 2
- 230000002411 adverse Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000007405 data analysis Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000000737 periodic effect Effects 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 238000013524 data verification Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000003631 expected effect Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/10—Pre-processing; Data cleansing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/213—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/243—Classification techniques relating to the number of classes
- G06F18/2433—Single-class perspective, e.g. one-against-all classification; Novelty detection; Outlier detection
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Artificial Intelligence (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The application discloses a threshold prediction method, a system, equipment and a storage medium, when the method is executed, statistical information corresponding to a target table is obtained and used for representing table self and information with variability in fields in the table, the statistical information corresponding to the target table is input into a prediction model to obtain a threshold result corresponding to the target table, the prediction model is trained based on historical statistical information and a preset machine learning algorithm and is iteratively optimized based on a prediction error, the prediction accuracy reaches a model with preset accuracy, and the prediction error is used for representing the difference between the predicted threshold result and an actual threshold in the training process. Compared with the prior art that the corresponding threshold value is determined by manually analyzing the table one by one in low efficiency, the scheme provided by the application can be used for inputting the statistical information corresponding to the table into the prediction model and further analyzing the statistical information by using the model to obtain the threshold value applicable to the current table, so that the efficient prediction of the threshold value is realized.
Description
Technical Field
The present disclosure relates to the field of computers, and in particular, to a threshold prediction method, system, device, and storage medium.
Background
In the current data analysis process, abnormality detection and identification are required to be carried out on the integral data quantity fluctuation or single field value fluctuation of the data set, so that adverse effects of abnormal data on analysis results are avoided.
The very important work in the data fluctuation verification is to determine a reasonable threshold value, the existing threshold value determination mainly depends on manual work, and along with the increase of the volume of data and the improvement of complexity, the process of manually determining the threshold value consumes longer and longer, and is not suitable for determining the threshold value for massive tables.
Therefore, how to determine the threshold value efficiently is a technical problem to be solved by those skilled in the art.
Disclosure of Invention
Based on the above problems, the present application provides a threshold prediction method, system, device and storage medium for efficiently predicting a threshold.
In order to solve the above problems, the technical solution provided in the embodiments of the present application is as follows:
the first aspect of the present application provides a threshold prediction method, including:
acquiring statistical information corresponding to a target table, wherein the statistical information is used for representing the table and information with variability in fields in the table;
inputting the statistical information corresponding to the target table into a prediction model to obtain a threshold value result corresponding to the target table, wherein the prediction model is trained based on historical statistical information and a preset machine learning algorithm, and is optimized based on prediction error iteration, the prediction accuracy reaches the preset accuracy, and the prediction error is used for representing the difference between the threshold value result obtained by prediction in the training process and an actual threshold value.
Optionally, the training process of the prediction model includes:
acquiring statistical information of a first preset table in a first preset time period;
determining a target feature based on the obtained statistical information;
dividing the target features into a training set and a testing set based on a preset proportion;
training the training set by using a preset machine learning algorithm to obtain a preparation model; evaluating the preparation model by using the test set, and adjusting the preparation model based on an evaluation result until the preparation model meets a preset standard to obtain a prediction model;
obtaining a prediction error corresponding to the prediction model;
and iteratively optimizing the prediction model based on the prediction error until the prediction model accords with a preset accuracy.
Optionally, the determining the target feature based on the obtained statistical information includes:
preprocessing the acquired statistical information to obtain preprocessed statistical information;
and processing the preprocessed statistical information based on a feature extraction algorithm to obtain target features.
Optionally, the method further comprises:
and storing the target statistical information and the threshold value result corresponding to the target statistical information into a data storage library, wherein the data storage library comprises statistical information and prediction results of each predicted table collected in a history mode, and the statistical information and the prediction results are used for providing data information for a training process and/or an iteration process of a prediction model.
Optionally, the obtaining the prediction error corresponding to the prediction model includes:
based on the prediction model, predicting statistical information corresponding to a preset period to obtain a threshold result, wherein the statistical information is information corresponding to a second preset table in a second preset time period corresponding to the preset period, and the statistical information is obtained from the data storage library;
acquiring an actual threshold value corresponding to a second preset table in the second preset time period;
an absolute or relative error between the predicted threshold result and the actual threshold is determined.
Optionally, the statistical information includes at least one of table number, data type of the field, value distribution of the field, table run time and table generation time.
Optionally, the target feature includes at least one of a business type feature, a time feature, and a location feature.
A second aspect of the present application provides a threshold prediction system, comprising:
the acquisition unit is used for acquiring statistical information corresponding to the target table, wherein the statistical information is used for representing the table and information with variability in fields in the table;
the prediction unit is used for inputting the statistical information corresponding to the target table into a prediction model to obtain a threshold value result corresponding to the target table, the prediction model is trained based on historical statistical information and a preset machine learning algorithm, and is optimized based on prediction error iteration, the prediction accuracy reaches the preset accuracy, and the prediction error is used for representing the difference between the threshold value result obtained by prediction in the training process and an actual threshold value.
A third aspect of the present application provides an electronic device, comprising: the apparatus comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the threshold prediction method of any one of the preceding first aspects when the computer program is executed.
A fourth aspect of the present application provides a computer readable storage medium having instructions stored therein which, when run on a terminal device, cause the terminal device to perform the threshold prediction method according to any one of the preceding first aspects.
Compared with the prior art, the application has the following beneficial effects:
according to the method, statistical information corresponding to the target table and used for representing the table and field information in the table is acquired, the statistical information corresponding to the target table is input into a prediction model, a threshold value result corresponding to the target table is obtained, the prediction model is trained based on historical statistical information and a preset machine learning algorithm, and is optimized based on prediction error iteration, the prediction accuracy reaches the model with preset accuracy, and the prediction error is used for representing the difference between the threshold value result obtained through prediction in the training process and an actual threshold value. Compared with the prior art that the corresponding threshold value is determined by manually analyzing the table one by one in low efficiency, the scheme provided by the application can be used for inputting the statistical information corresponding to the table into the prediction model and further analyzing the statistical information by using the model to obtain the threshold value applicable to the current table, so that the efficient prediction of the threshold value is realized.
Drawings
In order to more clearly illustrate the present embodiments or the technical solutions in the prior art, the drawings that are required for the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flowchart of a threshold prediction method according to an embodiment of the present application;
FIG. 2 is a schematic diagram of a model training process according to an embodiment of the present disclosure;
FIG. 3 is a block diagram of a threshold prediction system according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of a computer device according to an embodiment of the present application.
Detailed Description
In order to make the present application solution better understood by those skilled in the art, the following description will clearly and completely describe the technical solution in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.
In order to facilitate understanding of the technical solutions provided in the embodiments of the present application, the following description will first explain the background technology related to the embodiments of the present application.
As described above, the data fluctuation verification is one of important data verification methods, and has important significance for data analysis, decision and service operation, namely, abnormality detection and identification are performed on the whole data quantity fluctuation or single field value fluctuation of the data set, so as to avoid adverse effects of abnormal data on analysis results. In the process of data fluctuation verification, a very important task is to determine a reasonable threshold value, the current data fluctuation verification threshold value determination is mainly performed manually, and when the volume and complexity of managed data are rapidly increased, two major problems of threshold value accuracy and efficiency reduction are encountered: first, the accuracy is low, and existing schemes are manually set thresholds, which are often maximum, minimum, average, etc. However, the threshold value can change along with the characteristics of the service, the dimension of time, the region and other multiparty comprehensive factors, and the manual evaluation cannot adapt to the changes in time, so that the accuracy of the threshold value setting is low, the interference of data quality check on the developer is high, and the final data quality cannot be well ensured. Secondly, the manual work efficiency is low: the efficiency of obtaining the corresponding threshold value by manually analyzing the tables one by one is relatively low, and the method is not suitable for determining the threshold value for massive tables.
In order to solve the technical problems, the embodiment of the application provides a threshold prediction method, by collecting statistical information of a table, training historical fluctuation information by using a machine learning algorithm, the method can be more suitable for complex data such as periodic data, abnormal condition data and nonlinear data caused by service characteristics and time characteristics in a big data environment, and the accuracy of data fluctuation quality verification under the complex data condition is greatly improved by continuously generating online fluctuation information to update an algorithm model. In addition, even if machine learning is used, the accurate data fluctuation threshold value can be obtained rapidly in large quantities through engineering even if a large amount of data and massive statistical information are faced, compared with the existing manual work, the efficiency is greatly improved, and the labor cost is reduced.
In addition, the embodiment of the present application may also not limit the execution subject of the threshold prediction method, for example, the threshold prediction method of the embodiment of the present application may be applied to a data processing device such as a terminal device or a server. The terminal device may be an electronic device such as a smart phone, a computer, a Personal digital assistant (Personal DigitalAssistant, PDA), a tablet computer, etc. The server may be an independent server, a cloud server, or a cluster server composed of a plurality of servers.
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are only some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.
A threshold prediction method provided in the present application is described below by way of an embodiment. Referring to fig. 1, the flowchart of a threshold prediction method provided in the embodiment of the present application is shown, an execution body of the method flow is a server, and the method includes:
s101, acquiring statistical information corresponding to the target table.
The statistical information is used to characterize the table itself and information with variability in fields within the table. The statistical information of the target table is collected in an actual application scene, including but not limited to the information of the number of rows of the table, the data type of the fields, the value distribution of the fields (the numerical fields generally include the average value, the maximum value, the minimum value, the median and the like, the text fields include the number of enumerated values and the like), the running duration, the generation time and the like, so that the subsequent machine learning algorithm training and prediction are facilitated.
In one possible implementation, the statistical information includes at least one of a table number, a data type of a field, a value distribution of the field, a table run time, and a table generation time.
S102, inputting statistical information corresponding to the target table into a prediction model to obtain a threshold value result corresponding to the target table.
The prediction model is trained based on historical statistical information and a preset machine learning algorithm, and is optimized based on prediction error iteration, the prediction accuracy reaches the preset accuracy, and the prediction error is used for representing the difference between a predicted threshold result and an actual threshold in the training process.
In one possible implementation, the training process of the prediction model includes the following steps A1-A6:
and A1, acquiring statistical information of a first preset table in a first preset time period.
The first preset table is a table which needs to be predicted currently, and the first preset time period can be adjusted adaptively according to actual requirements.
And A2, determining target characteristics based on the acquired statistical information.
In an actual application scene, a feature extraction algorithm can be selected to extract the target features.
In one possible implementation, the target feature includes at least one of a business type feature, a time feature, and a place feature.
In a possible implementation manner, the determining the target feature based on the obtained statistical information includes step B1-step B2:
and step B1, preprocessing the acquired statistical information to obtain preprocessed statistical information.
The collected related information is preprocessed, and the preprocessing can comprise at least one of data cleaning, abnormal value detection and processing and standardization.
And step B2, processing the preprocessed statistical information based on a feature extraction algorithm to obtain target features.
And extracting key features from the preprocessed data by using a feature extraction algorithm. The target features extracted in the step comprise, but are not limited to, business characteristic features of different dangerous types in the insurance industry, time features, place features and the like of holidays such as national celebration of promotion date and the like.
And A3, dividing the target features into a training set and a testing set based on a preset proportion.
The data after feature extraction is divided into a training set and a testing set according to a certain proportion. The preset proportion can be adaptively adjusted according to actual requirements.
Step A4, training the training set by using a preset machine learning algorithm to obtain a preparation model; and evaluating the preparation model by using the test set, and adjusting the preparation model based on an evaluation result until the preparation model meets a preset standard to obtain a prediction model.
And training the training set by using a proper machine learning algorithm to obtain a corresponding preparation model. And evaluating the trained preparation model by using the test data set to obtain an evaluation result, wherein the evaluation result corresponds to the difference between the prediction result of the comparison model and the actual data, and when the preparation model has stable performance on the training set and the test set and reaches the expected effect, the preparation model meets the preset standard at the moment, and the preparation model at the moment is output to obtain the prediction model. Wherein the machine learning algorithm may be a neural network or the like.
And step A5, obtaining a prediction error corresponding to the prediction model.
In one possible implementation manner, the obtaining the prediction error corresponding to the prediction model includes step C1 to step C3:
and step C1, based on the prediction model, predicting statistical information corresponding to a preset period to obtain a threshold result.
The statistical information is obtained from the data storage library, and the statistical information is information corresponding to a second preset table in a second preset time period corresponding to the preset period.
The prediction model obtained after training is deployed in an online environment in the above step, that is, the prediction model is applied to predict the data table, wherein the preset period can be day, week and month, and after the corresponding preset period is determined, the data fluctuation threshold value in a future period (that is, in a second preset period) is predicted based on the preset period. The prediction target is a second preset table, and the second table is used for determining a difference between the current model prediction effect and an actual threshold value, and the second preset table may be a table which is selected by a system operator and has been predicted before, or may be a table which has not been predicted and needs to be subjected to a subsequent threshold value.
And C2, acquiring an actual threshold value corresponding to a second preset table in the second preset time period.
That is, the actual threshold value of the prediction target is obtained, and the actual threshold value may be manually confirmed or determined based on other algorithms, and is used as a reference for comparison.
And C3, determining an absolute error or a relative error between the predicted threshold result and the actual threshold.
The absolute error is the absolute value of the difference between the measured value and the true value, i.e., absolute error= |measured value-true value|; the relative error is the percentage of the absolute error to the true value, i.e., the relative error= |measurement-true value|/true value. The absolute error is a value indicating both the magnitude and the positive and negative directions of the error, reflecting the magnitude of the deviation of the measurement result from the true value in the same unit dimension, and it represents exactly the actual magnitude of the deviation from the true value.
And step A6, iteratively optimizing the prediction model based on the prediction error until the prediction model accords with a preset accuracy rate.
And after the algorithm model is trained, establishing online data fluctuation threshold prediction capability, and predicting the data fluctuation threshold of the next stage according to the historical fluctuation information condition. After the machine learning algorithm predicts the threshold, data is saved, actual data fluctuation information of the predicted table is saved, and the algorithm model is iterated through the data regularly.
In one possible implementation, the method further includes the steps of:
s103, storing the target statistical information and a threshold result corresponding to the target statistical information into a data storage library.
The data repository includes historically collected statistics and predictions for each predicted table to provide data information for training and/or iterating a predictive model.
That is, in the step of acquiring the statistics corresponding to the target table in S101, the acquired statistics may be backed up to the data repository, and this step may be performed while acquiring the statistics, or may be performed after the processing based on the statistics is completed.
The step is used for carrying out data retention on the prediction threshold value of the prediction model and the fluctuation information actually generated by the predicted table, so that backtracking and inquiry of subsequent data are facilitated, iterative optimization is carried out on the algorithm model through the data, and the accuracy of data fluctuation verification is further improved. Referring to fig. 2, fig. 2 is a schematic diagram of a model training flow provided in an embodiment of the present application, where the left side is a process of performing machine learning algorithm training based on table statistics collected by history in the present application, and the right side is a process of performing further prediction iterative optimization on a model generated in the foregoing step in the present application, that is, performing optimization iteration on the model by using statistics information of the model obtained by training in an actual application environment and a threshold generated by prediction.
In summary, the application trains the historical fluctuation information by collecting the fluctuation condition of the data and using the machine learning algorithm, and the application can collect and save the statistical information of the table, including but not limited to the information of the number of rows of the table, the data type of the field, the value distribution of the field, the operation time, the output time and the like, so that the subsequent machine learning algorithm training and prediction are facilitated. Training the historical volatility information of the table using a machine learning algorithm: the method and the device use a machine learning algorithm (including but not limited to an algorithm model such as a time sequence model and a neural network algorithm) to train the acquired fluctuation information, and the greater the acquired fluctuation information time range is, the higher the accuracy of a related algorithm is. In the machine learning algorithm training, the method combines various factors such as business characteristics, data characteristics, time characteristics and the like, and improves the accuracy of the training model.
Therefore, the method can be more suitable for complex data such as periodic data, abnormal condition data, nonlinear data and the like caused by service characteristics and time characteristics in a big data environment, and the accuracy of data fluctuation quality verification under the complex data condition is greatly improved through continuously generating online fluctuation information to update an algorithm model in an iterative manner. Even if machine learning is used, accurate data fluctuation threshold values can be obtained rapidly in large quantities through engineering even if a large amount of data and massive statistical information are faced, compared with the existing manual work, the efficiency is greatly improved by making the data fluctuation threshold values one by one, and the labor cost is reduced.
The above is some specific implementations of the threshold prediction method provided in the embodiments of the present application, and based on this, the present application further provides a corresponding system for threshold prediction. The system provided in the embodiments of the present application will be described from the viewpoint of functional modularization. Fig. 3 is a block diagram of a threshold prediction system according to an embodiment of the present application.
The system comprises:
an obtaining unit 110, configured to obtain statistics information corresponding to the target table, where the statistics information is used to characterize the table itself and information with variability in fields in the table;
the prediction unit 111 is configured to input statistical information corresponding to the target table into a prediction model, and obtain a threshold result corresponding to the target table, where the prediction model is trained based on historical statistical information and a preset machine learning algorithm, and is iteratively optimized based on a prediction error, and the prediction accuracy reaches a preset accuracy, and the prediction error is used to characterize a difference between a threshold result obtained by prediction in a training process and an actual threshold.
Optionally, the training process of the prediction model includes:
acquiring statistical information of a first preset table in a first preset time period;
determining a target feature based on the obtained statistical information;
dividing the target features into a training set and a testing set based on a preset proportion;
training the training set by using a preset machine learning algorithm to obtain a preparation model;
evaluating the preparation model by using the test set, and adjusting the preparation model based on an evaluation result until the preparation model meets a preset standard to obtain a prediction model;
obtaining a prediction error corresponding to the prediction model;
and iteratively optimizing the prediction model based on the prediction error until the prediction model accords with a preset accuracy.
Optionally, the system further comprises:
the storage unit is used for storing the target statistical information and the threshold value result corresponding to the target statistical information into a data storage library, wherein the data storage library comprises statistical information and prediction results of each predicted table collected in a history mode, and the data storage library is used for providing data information for a training process and/or an iteration process of a prediction model.
Optionally, the statistical information includes at least one of table number, data type of the field, value distribution of the field, table run time and table generation time.
Optionally, the target feature includes at least one of a business type feature, a time feature, and a location feature.
The embodiment of the application also provides corresponding equipment and a computer storage medium, which are used for realizing the scheme provided by the embodiment of the application.
As shown in fig. 4, the computer device 01 is in the form of a general purpose computing device. The components of the computer device 01 may include, but are not limited to: one or more processors or processing units 03, a system memory 08, and a bus 04 that connects the various system components (including the system memory 08 and processing units 03).
Bus 04 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, and a local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, micro channel architecture (MAC) bus, enhanced ISA bus, video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.
The computer device 01 typically includes a variety of computer system readable media. Such media can be any available media that can be accessed by the computer device 01 and includes both volatile and nonvolatile media, removable and non-removable media.
The system memory 08 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM) 09 and/or cache memory 10. The computer device 01 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 11 may be used to read from or write to non-removable, nonvolatile magnetic media (not shown in FIG. 4, commonly referred to as a "hard disk drive"). Although not shown in fig. 4, a magnetic disk drive for reading from and writing to a removable non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading from or writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be coupled to bus 04 through one or more data medium interfaces. The memory 08 may include at least one program product having a set (e.g., at least one) of program modules configured to carry out the functions of the embodiments of the present application.
A program/utility 12 having a set (at least one) of program modules 13 may be stored in, for example, memory 08, such program modules 13 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment. Program modules 13 generally perform the functions and/or methods in the embodiments described herein.
The computer device 01 may also communicate with one or more external devices 02 (e.g., keyboard, pointing device, display 07, etc.), one or more devices that enable a user to interact with the computer device 01, and/or any devices (e.g., network card, modem, etc.) that enable the computer device 01 to communicate with one or more other computing devices. Such communication may be through an input/output (I/O) interface 06. Moreover, the computer device 01 may also communicate with one or more networks such as a Local Area Network (LAN), a Wide Area Network (WAN) and/or a public network, such as the Internet, through the network adapter 05. As shown in fig. 4, the network adapter 05 communicates with other modules of the computer device 01 via the bus 04. It should be appreciated that although not shown in fig. 4, other hardware and/or software modules may be used in connection with the computer device 01, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data backup storage systems, and the like.
The processor unit 03 executes various functional applications and data processing by running programs stored in the system memory 08, for example, to implement a front-end button authority management method provided in the embodiments of the present application.
From the above description of embodiments, it will be apparent to those skilled in the art that all or part of the steps of the above described example methods may be implemented in software plus general hardware platforms. Based on such understanding, the technical solutions of the present application may be embodied in the form of a software product, which may be stored in a storage medium, such as a read-only memory (ROM)/RAM, a magnetic disk, an optical disk, or the like, including several instructions for causing a computer device (which may be a personal computer, a server, or a network communication device such as a router) to perform the methods described in the embodiments or some parts of the embodiments of the present application.
It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
It should be further noted that, in the present specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for the apparatus and device embodiments, since they are substantially similar to the method embodiments, the description is relatively simple, and reference is made to the description of the method embodiments for relevant points. The apparatus and device embodiments described above are merely illustrative, wherein elements illustrated as separate elements may or may not be physically separate, and elements presented as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
The foregoing is merely one specific embodiment of the present application, but the protection scope of the present application is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present application should be covered in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (10)
1. A method of threshold prediction, comprising:
acquiring statistical information corresponding to a target table, wherein the statistical information is used for representing the table and information with variability in fields in the table;
inputting the statistical information corresponding to the target table into a prediction model to obtain a threshold value result corresponding to the target table, wherein the prediction model is trained based on historical statistical information and a preset machine learning algorithm, and is optimized based on prediction error iteration, the prediction accuracy reaches the preset accuracy, and the prediction error is used for representing the difference between the threshold value result obtained by prediction in the training process and an actual threshold value.
2. The method of claim 1, wherein the training process of the predictive model comprises:
acquiring statistical information of a first preset table in a first preset time period;
determining a target feature based on the obtained statistical information;
dividing the target features into a training set and a testing set based on a preset proportion;
training the training set by using a preset machine learning algorithm to obtain a preparation model;
evaluating the preparation model by using the test set, and adjusting the preparation model based on an evaluation result until the preparation model meets a preset standard to obtain a prediction model;
obtaining a prediction error corresponding to the prediction model;
and iteratively optimizing the prediction model based on the prediction error until the prediction model accords with a preset accuracy.
3. The method of claim 2, wherein the determining the target feature based on the obtained statistical information comprises:
preprocessing the acquired statistical information to obtain preprocessed statistical information;
and processing the preprocessed statistical information based on a feature extraction algorithm to obtain target features.
4. The method according to claim 1, wherein the method further comprises:
and storing the target statistical information and the threshold value result corresponding to the target statistical information into a data storage library, wherein the data storage library comprises statistical information and prediction results of each predicted table collected in a history mode, and the statistical information and the prediction results are used for providing data information for a training process and/or an iteration process of a prediction model.
5. The method of claim 4, wherein the obtaining the prediction error corresponding to the prediction model comprises:
based on the prediction model, predicting statistical information corresponding to a preset period to obtain a threshold result, wherein the statistical information is information corresponding to a second preset table in a second preset time period corresponding to the preset period, and the statistical information is obtained from the data storage library;
acquiring an actual threshold value corresponding to a second preset table in the second preset time period;
an absolute or relative error between the predicted threshold result and the actual threshold is determined.
6. The method of claim 1, wherein the statistical information comprises at least one of table number, data type of field, value distribution of field, table run time, and table generation time.
7. The method of claim 2, wherein the target characteristics include at least one of a business type characteristic, a time characteristic, and a location characteristic.
8. A threshold prediction system, the system comprising:
the acquisition unit is used for acquiring statistical information corresponding to the target table, wherein the statistical information is used for representing the table and information with variability in fields in the table;
the prediction unit is used for inputting the statistical information corresponding to the target table into a prediction model to obtain a threshold value result corresponding to the target table, the prediction model is trained based on historical statistical information and a preset machine learning algorithm, and is optimized based on prediction error iteration, the prediction accuracy reaches the preset accuracy, and the prediction error is used for representing the difference between the threshold value result obtained by prediction in the training process and an actual threshold value.
9. An electronic device, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the threshold prediction method of any of claims 1-7 when the computer program is executed.
10. A computer readable storage medium, characterized in that the computer readable storage medium has stored therein instructions, which when run on a terminal device, cause the terminal device to perform the threshold prediction method according to any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311501716.5A CN117473268A (en) | 2023-11-10 | 2023-11-10 | Threshold prediction method, system, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311501716.5A CN117473268A (en) | 2023-11-10 | 2023-11-10 | Threshold prediction method, system, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117473268A true CN117473268A (en) | 2024-01-30 |
Family
ID=89631075
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311501716.5A Pending CN117473268A (en) | 2023-11-10 | 2023-11-10 | Threshold prediction method, system, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117473268A (en) |
-
2023
- 2023-11-10 CN CN202311501716.5A patent/CN117473268A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111045894B (en) | Database abnormality detection method, database abnormality detection device, computer device and storage medium | |
| CN107491970B (en) | Real-time anti-cheating detection monitoring method and system and computing equipment | |
| CN110059894B (en) | Equipment state evaluation method, device, system and storage medium | |
| CN110264270B (en) | Behavior prediction method, behavior prediction device, behavior prediction equipment and storage medium | |
| CN113837596B (en) | Fault determination method and device, electronic equipment and storage medium | |
| CN113312578B (en) | Fluctuation attribution method, device, equipment and medium of data index | |
| CN113538154A (en) | Risk object identification method and device, storage medium and electronic equipment | |
| CN110083518B (en) | AdaBoost-Elman-based virtual machine software aging prediction method | |
| CN111242779B (en) | Financial data characteristic selection and prediction method, device, equipment and storage medium | |
| RU2632124C1 (en) | Method of predictive assessment of multi-stage process effectiveness | |
| CN116774986A (en) | Automatic evaluation method and device for software development workload, storage medium and processor | |
| CN111783883A (en) | Abnormal data detection method and device | |
| CN116957828A (en) | Method, equipment, storage medium and device for checking account | |
| CN111382909A (en) | Rejection inference method based on survival analysis model expansion bad sample and related equipment | |
| CN117473268A (en) | Threshold prediction method, system, equipment and storage medium | |
| CN110807082A (en) | Quality spot check item determination method, system, electronic device and readable storage medium | |
| CN115204501A (en) | Enterprise evaluation method and device, computer equipment and storage medium | |
| CN113987351A (en) | Artificial intelligence based intelligent recommendation method and device, electronic equipment and medium | |
| CN113610225A (en) | Quality evaluation model training method and device, electronic equipment and storage medium | |
| CN112949951A (en) | Data prediction method, data prediction device, electronic equipment and storage medium | |
| CN112950392A (en) | Information display method, posterior information determination method and device and related equipment | |
| CN116149971B (en) | Equipment fault prediction method and device, electronic equipment and storage medium | |
| CN113704103B (en) | Test case recommendation method and device, medium and electronic equipment | |
| CN111626586B (en) | Data quality detection method, device, computer equipment and storage medium | |
| CN116797010A (en) | Work number identification method, device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |