CN108090138A - The monitoring method and system of a kind of data warehouse - Google Patents
The monitoring method and system of a kind of data warehouse Download PDFInfo
- Publication number
- CN108090138A CN108090138A CN201711230833.7A CN201711230833A CN108090138A CN 108090138 A CN108090138 A CN 108090138A CN 201711230833 A CN201711230833 A CN 201711230833A CN 108090138 A CN108090138 A CN 108090138A
- Authority
- CN
- China
- Prior art keywords
- data
- increment
- tables
- time series
- time period
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/283—Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The present invention provides a kind of monitoring method of data warehouse, including：S1 obtains any data table in data warehouse, and integrating moving average model according to autoregression predicts data increment of the tables of data in the fixed time period of prediction day, and the data increment is the increase item number of data in tables of data；S2, calculates real data increment of the prediction day in fixed time period, and the real data increment with the data increment predicted is compared, monitors whether the tables of data carries out early warning.The present invention is compared by the way that the actual increment of the data gathered from data warehouse to be integrated to the increment of moving average model prediction with autoregression, early warning is carried out in actual increment and excessive prediction increment deviation, and realize and comprehensive monitoring is carried out to the data volume of each tables of data in data warehouse, it can be with the stability of effective guarantee data warehouse.
Description
Technical field
The present invention relates to database processing technical field, a kind of monitoring method more particularly, to data warehouse and it is
System.
Background technology
One data warehouse corresponds to multiple business datum sources.With business deepen constantly, it is necessary to the data analyzed not yet
It is disconnected to increase, correspondingly, data warehouse task is various.There are substantial amounts of newly-increased data to be stored in data warehouse daily, if daily
Occur exception in task processes, the quality of data can be influenced, and the data in the next time backward may be generated
It influences.Therefore exception of the data in processing procedure is found in time, is handled.Data warehouse is very important.
A kind of method of data warehouse data monitoring is provided in the prior art, by performing timing monitor task and correlation
A series of configurations can be achieved to be monitored the data cases of data warehouse newer table daily, can also realize because of business number
According to source change and will output exception early warning, can find the various problems in program emerged in operation in time.
A kind of database monitoring method is additionally provided in the prior art, according to the information of each database, is obtained and each data
The corresponding monitoring configuration file of information in storehouse；According to acquisition monitoring configuration file corresponding with the information of each database, utilize
The monitoring programme corresponding with the information of each database write in advance, is monitored corresponding database.
But the prior art business routine data amount is not changed and may output exception early warning and to data
The data volume of each tables of data in warehouse carries out comprehensive monitoring, can not effective guarantee data warehouse stability.
The content of the invention
The present invention provides a kind of monitoring for the data warehouse for overcoming the above problem or solving the above problems at least partly
Method and system.
On the one hand, the present invention provides a kind of monitoring method of data warehouse, including：
S1 obtains any data table in data warehouse, and integrating moving average model according to autoregression predicts the data
Data increment of the table in the fixed time period of prediction day, the data increment are the increase item number of data in tables of data；
S2 calculates real data increment of the prediction day in fixed time period, by the real data increment and in advance
The data increment surveyed is compared, and monitors whether the tables of data carries out early warning.
Preferably, include before step S1：
Database name and data table name in the data warehouse, search any data in the data warehouse
Table.
Preferably, the autoregression integration moving average model includes autoregression model, moving average model and autoregression
Moving average model, the integration moving average model of autoregression described in step S1 are established by following steps：
The data increment of the fixed time period in described prediction continuous several weeks a few days ago is formed into time series, when will be described
Between series processing be stationary time series；
According to the partial autocorrelation function and auto-correlation function of the stationary time series, establish and the stationary time series
Corresponding autoregression integrates moving average model.
Preferably, it is described to include the time Series Processing for the specific steps of stationary time series：
It is nonstationary time series according to the time series, one is carried out to each data increment in the time series
Secondary calculus of differences obtains new time series；
Judge whether the new time series is stationary time series, if it is not, previous step is then returned, until new
Time series is stationary time series.
Preferably, the partial autocorrelation function and auto-correlation function according to the stationary time series, establish with it is described
The corresponding autoregression integration moving average model of stationary time series specifically includes：
If the partial autocorrelation function is with truncation and auto-correlation function is with hangover property, establish and the stationary time
The corresponding autoregression model of sequence；
If partial autocorrelation function is with hangover property and auto-correlation function is with truncation, establish and the stationary time series
Corresponding moving average model；
If partial autocorrelation function and auto-correlation function are respectively provided with hangover, foundation is corresponding with the stationary time series certainly
Regressive averaging model；
Wherein, the determination methods of the hangover property are function by exponential damping or into sine wave, the truncation
Determination methods are that functional value is zero.
Preferably, fixed time period is obtained by following steps described in step S1：
Multiple special time periods were chosen in one week, gather the tables of data each specific time within continuous several weeks
Data increment in section calculates the variance of the data increment in each special time period；
Compare the variance of data increment in all special time periods, using the special time period of variance minimum as described in
Fixed time period.
Preferably, real data increment is obtained by following steps described in step S2：
Record the write time in each tables of data per data in the data warehouse；It is calculated by the said write time
The data increment of tables of data in the fixed time period.
Preferably, the real data increment for calculating each prediction day in fixed time period specifically includes：
By the said write time, the primary data total amount of the tables of data when fixed time period starts is gathered, with
And at the end of the fixed time period tables of data end total amount of data；
The end total amount of data is subtracted into the primary data total amount, obtains the tables of data in the fixed time period
Data increment.
Preferably, the real data increment with the data increment predicted is compared described in step S2, supervised
Controlling the tables of data and whether carrying out the specific steps of early warning includes：
The ratio of the real data increment and the prediction data increment is subtracted 1 as deviation, if the deviation
Absolute value be more than 10%, to the tables of data carry out early warning.
On the other hand, the present invention provides a kind of monitoring system of data warehouse, including：
For obtaining any data table in data warehouse, sliding average mould is integrated according to autoregression for incremental forecasting module
Type predicts data increment of the tables of data in the fixed time period of prediction day, and the data increment is data in tables of data
Increase item number；
Increment comparing module, for calculating real data increment of the prediction day in fixed time period, by the reality
Border data increment is compared with the data increment predicted, monitors whether the tables of data carries out early warning.
The monitoring method and system of a kind of data warehouse provided by the invention, pass through the data that will be gathered from data warehouse
The increment of actual increment and autoregression integration moving average model prediction be compared, in actual increment with predicting increment deviation
Early warning is carried out when excessive, and realizes and comprehensive monitoring is carried out to the data volume of each tables of data in data warehouse, it can
With the stability of effective guarantee data warehouse.
Description of the drawings
Fig. 1 is the flow chart of the monitoring method of the data warehouse of one embodiment of the invention；
Fig. 2 is the structure diagram of the monitoring system of the data warehouse of one embodiment of the invention.
Specific embodiment
With reference to the accompanying drawings and examples, the specific embodiment of the present invention is described in further detail.Implement below
Example is not limited to the scope of the present invention for illustrating the present invention.
It is the flow chart of the monitoring method of the data warehouse of one embodiment of the invention referring to Fig. 1, including：S1 obtains number
According to any data table in warehouse, fixed time period of the moving average model prediction data table in prediction day is integrated according to autoregression
Interior data increment, data increment are the increase item number of data in tables of data；S2 calculates reality of the prediction day in fixed time period
Real data increment is compared border data increment with the data increment predicted, whether monitoring data table carries out early warning.
Specifically, autoregression integration moving average model is a kind of common time series predicting model, can be pre- by inciting somebody to action
The data sequence that survey object is formed over time is considered as a random time series, then again with certain mathematical model
Carry out approximate description this time series, this model be identified it is errorless after, so that it may from the past value of time series and value is come now
Predict future value, the time series data predicted.The embodiment of the present invention by obtaining any one tables of data in data warehouse,
And according to autoregression integrate moving average model come the data in prediction data table prediction day some fixed time period in
Data increment obtains the prediction data increment of prediction day.Here prediction refers to any one or more futures to be predicted day
Date, data increment then refer to the increase item number of the data in tables of data in fixed time period.
Total amount of data of the above-mentioned tables of data before and after the fixed time period of prediction day is detected, and the tables of data is calculated and exists
The real data increment in the fixed time period of day is predicted, by real data increment and autoregression integration moving average model prediction
Obtained prediction data increment is compared, and when deviation between the two is more than default difference, which is carried out pre-
It is alert, it is whether abnormal by the data volume variation in any one tables of data in monitoring data warehouse, to realize to entire data bins
The monitoring in storehouse.Wherein, alarm mode includes sending mail or SMS to relevant staff, but is not limited to
This.
The embodiment of the present invention is flat by the way that the actual increment of the data gathered from data warehouse and autoregression integration are slided
The increment of equal model prediction is compared, and carries out early warning in actual increment and excessive prediction increment deviation, and realizes logarithm
Comprehensive monitoring is carried out according to the data volume of each tables of data in warehouse, it can be with the stability of effective guarantee data warehouse.
Based on above-described embodiment, as a kind of optional embodiment, database name and tables of data in data warehouse
, any data table in searching data warehouse.
Specifically, the data that data warehouse is a subject-oriented, integrated, metastable, reflecting history changes
Set.Data warehouse is a data source active set to multiple isomeries towards analytic type data processing for supporting decision-making
Into.Mass data is store in the database of data warehouse, is on the basis that original scattered database data is extracted, cleared up
On process, summarize and arrange by system, by different tables of data by these data classification ensembles in data warehouse
In.Due to the table name of any data table in the database be all it is unique existing, database name and tables of data can be passed through
Name, unique tables of data is found from data warehouse, any one tables of data in data warehouse can be found accordingly, and it is carried out
Monitoring.
Based on above-described embodiment, as a kind of optional embodiment, autoregression integration moving average model includes autoregression
Model, moving average model and autoregressive moving-average model, autoregression integration moving average model passes through following in step S1
Step is established：The data increment for predicting fixed time period in continuous several weeks a few days ago is formed into time series, by time Series Processing
For stationary time series；According to the partial autocorrelation function and auto-correlation function of stationary time series, foundation and stationary time series
Corresponding autoregression integrates moving average model.
Specifically, there are a fixed time period, which can be weekly the working day of the week,
Can be specific certain day 7 points to 8 points of evening etc. weekly for each week it is fixed existing for a period.Obtain prediction
Data increment of the tables of data in this fixed time period in continuous several weeks a few days ago, above-mentioned data increment have it is multiple, by institute
Some data increments chronologically form a time series, the processing of calculus of differences are carried out to the time series, after being handled
Stationary time series.According to the auto-correlation function and partial autocorrelation function of the stationary time series, can establish steady with this
The corresponding autoregression integration moving average model of time series, i.e. autoregression model, moving average model and autoregression is slided flat
One kind in this three class model of equal model.Wherein, auto-correlation function is the phase relation ordered series of numbers of stochastic variable and its lagged variable, is used
To investigate the correlation intensity of stochastic variable and its lagged variable, in embodiments of the present invention, stochastic variable refers to form time sequence
Each data increment of row, and lagged variable refers to from sequential than the variable of current variable hysteresis, for example, NtRefer to data
In warehouse in certain tables of data yesterday data increment, and Nt-1 then represents the data increment of the day before yesterday, then it is stagnant to be referred to as monovalence by Nt-1
Variable afterwards.It should be noted that partial autocorrelation function is another method for describing random process structure feature.
Based on above-described embodiment, as a kind of optional embodiment, calculus of differences is carried out to time series, when obtaining steady
Between the specific steps of sequence include：It is nonstationary time series according to time series, to each data increment in time series
First difference computing is carried out, obtains new time series；Judge whether new time series is stationary time series, if it is not,
Previous step is then returned to, until new time series is stationary time series.
Specifically, judge whether the time series that above-mentioned all data increments are chronologically formed is stationary time series,
So-called stationary time series refers to that time series data is stable, without trend and periodic time series, the i.e. time series
In the averages of all time series datas possess the amplitude of constant on a timeline, and the variance of time series data tends to be same on a timeline
All statistical natures of one stationary value, the i.e. time series are all the constants on the time.When above-mentioned data increment chronologically
When the time series of composition is nonstationary time series, calculus of differences, fortune are carried out to each data increment in time series
A new time series is obtained after calculation.Judge whether new time series is stationary time series, if it is not, then continuing
Calculus of differences is carried out to the data in the time series, new time series is generated again, until newly-generated time series is
Until stabilization time sequence.
It should be noted that the time series for establishing autoregression integration moving average model must be stable, non-stationary
Time series can not capture rule, such as the stock certificate data for usually being influenced and being fluctuated by policy and news, due to the number of share of stock
According to being non-stable, therefore moving average model can not be integrated with autoregression to predict following stock certificate data.
Based on above-described embodiment, as a kind of optional embodiment, according to the partial autocorrelation function of stationary time series and
Auto-correlation function is established autoregression integration moving average model corresponding with stationary time series and is specifically included：If partial autocorrelation
Function is with truncation and auto-correlation function is with hangover property, establishes autoregression model corresponding with stationary time series；If partially
Auto-correlation function is with hangover property and auto-correlation function is with truncation, establishes sliding average mould corresponding with stationary time series
Type；If partial autocorrelation function and auto-correlation function are respectively provided with hangover, establish autoregression corresponding with stationary time series and slide
Averaging model；The determination methods of hangover property press exponential damping or into sine wave for function, and the determination methods of truncation are letter
Numerical value is zero.
Specifically, the stationary time series obtained according to above-described embodiment, Observable are corresponding with the stationary time series
Auto-correlation function and partial autocorrelation function establish corresponding autoregression integration moving average model, i.e. autoregression model, slip is flat
One kind in equal model and autoregressive moving-average model this three class model.
When partial autocorrelation function with truncation and auto-correlation function with hangover property when, establish and stationary time series pair
The autoregression model answered, autoregression model are with the model for itself doing regression variable, that is, utilize the random change at early period at several moment
The linear combination of amount is a kind of common shape in time series come the linear regression model (LRM) of certain moment stochastic variable after describing
Formula can establish p rank autoregression model AR (p) by acquiring the hysteresis number p of the time series data used in prediction model in itself；When
Partial autocorrelation function is with hangover property and auto-correlation function is with truncation, establishes sliding average corresponding with stationary time series
Model, moving average model are a kind of according to time series data, item by item elapse, and are put down when calculating the sequence comprising certain item number successively
Average to reflect the model of long-term trend, the hysteresis number q of the prediction error used in prediction model can be represented by acquiring to build
Vertical q rank moving average model MA (q)；If partial autocorrelation function and auto-correlation function are respectively provided with hangover, foundation and stationary time
The corresponding autoregressive moving-average model of sequence, autoregressive moving-average model are the important methods of search time sequence, by certainly
Regression model is formed with being mixed based on moving average model, it is necessary to acquire the time series data for representing and being used in prediction model in itself
Hysteresis number p and represent the hysteresis number q of the prediction error used in prediction model, to establish autoregressive moving-average model ARMA
(p, q).
It should be noted that for partial autocorrelation function and auto-correlation function, the determination methods of hangover property are function
By exponential damping or into sine wave, and it is zero that the determination methods of truncation, which are functional value,.
Based on above-described embodiment, as a kind of optional embodiment, fixed time period is obtained by following steps in step S1
It takes：Multiple special time periods were chosen in one week, data of the gathered data table within continuous several weeks in each special time period increase
Amount calculates the variance of the data increment in each special time period；Compare the variance of data increment in whole special time periods, it will
The special time period of variance minimum is as fixed time period.
Specifically, since the content that is stored in each tables of data is different, weekly in different period to each tables of data
The influence of middle data increment is also different.For example, when the sales volume of beer described in some tables of data, due to working day with
The difference of nonworkdays, more people can select to drink to loosen at weekend, therefore sales volume of the beer at weekend has apparent increasing
Add, working day, daily data increment then had apparent difference compared with weekend in tables of data at this time, continuous more data in the daytime
Increment then has big difference, and has periodic variation.Some fixed time period is selected, and should in continuous several weeks a few days ago according to predicting
, it is necessary to choose the data in fixed time period when data increment in fixed time period establishes autoregression integration moving average model
Increment more they tends to stabilization, i.e., variance is smaller.
Therefore, multiple representative specific times that may be impacted to data increment were first chosen from one week
Section, for example, can using working day or weekend daily as a special time period, can also by 9 points to 10 points of every morning,
7 points to 9 points of 2 points to 5 points of every afternoon or every night etc. daily a period of time as a special time period, for choosing
Fixed each special time period, the data increment in gathered data table special time period all within continuous several weeks, and calculate
The variance of these data increments.For example, 9 points of working day weekly, weekend and every morning are chosen respectively to 12 points as three
Special time period, and the data increment in continuous surrounding in three special time periods is obtained respectively, acquire work weekly in surrounding
Make the variance of daily data increment in day as first variance, the variance of daily data increment is second variance in weekend weekly,
9 points of variances to 12 point data increments of every morning are poor for third party, are respectively compared first variance, second variance and third party
The size of difference, if finding, first variance is more than second variance, and first variance is poor less than third party, it is known that and third party's difference is minimum,
Then the data increment in 9 points to 12 points this special time periods of every morning more they tends to stable state, therefore, selects every morning
9 points to 12 points are used as fixed time period, and the autoregression integration moving average model established with data increment in the period is predicted
Data increment it is more accurate and more representative.
Based on above-described embodiment, as a kind of optional embodiment, real data increment passes through following steps in step S2
It obtains：Record the write time in each tables of data per data in data warehouse；Fixed time period is calculated by the write time
The data increment of interior tables of data.
Specifically, when data write the tables of data in data warehouse, the write time per data is recorded, so as to basis
The write time of data judges the total amount of data in data warehouse in any one tables of data in each period, calculates accordingly solid
The data increment of any one tables of data in section of fixing time.
Based on above-described embodiment, as a kind of optional embodiment, actual number of the prediction day in fixed time period is calculated
It is specifically included according to increment：By the write time, gather the primary data total amount of tables of data when fixed time period starts and fix
The end total amount of data of tables of data at the end of period；It will terminate total amount of data and subtract primary data total amount, obtain the set time
The data increment of tables of data in section.
Specifically, write time during tables of data is write according to data, period data when starting can be fixed
The total amount of all data is as primary data total amount in table, and by the tables of data at the end of fixed time period all data it is total
Amount is as end total amount of data, it is known that data increment at this time is terminates the difference between total amount of data and primary data total amount, i.e.,
It is the total amount increased or decreased from the data in tables of data in fixed time period start to finish this period.
Based on above-described embodiment, as a kind of optional embodiment, by real data increment and the number of prediction in step S2
It is compared according to increment, the specific steps whether monitoring data table carries out early warning include：By real data increment and the number of prediction
Subtract 1 as deviation according to the ratio of increment, if the absolute value of deviation is more than 10%, early warning is carried out to tables of data.
Specifically, the data increment real data increment measured and autoregression integration moving average model predicted
It is compared, calculates the size of the deviation of the two, know whether the tables of data should carry out early warning.By real data increment with
The ratio of the data increment of prediction subtracts 1, you can the size of the deviation of the two is obtained, when the absolute value of deviation is more than threshold value
When 10%, it is known that the deviation between real data increment and prediction data increment is excessive, then carries out early warning, Xiang Xiang to the tables of data
The staff of pass sends warning information.It should be noted that when threshold value is too small, since there are multiple data in data warehouse
Table, and each tables of data has different degrees of data volume update daily, threshold value is set too small, may cause data warehouse frequency
The phenomenon that numerous early warning；And when threshold value is excessive, there may be the data in tables of data to generate missing and omit but do not carry out early warning
Therefore measure, sets the threshold to 10%, can more effectively ensure the stability of database.
The optimum modeling period of any data table establishes autoregression integration and slides in selected data of embodiment of the present invention warehouse
Dynamic averaging model, and the actual increment of the data gathered from data warehouse and autoregression integration moving average model are predicted
Increment is compared, and carries out early warning in actual increment and excessive prediction increment deviation, realizes to each in data warehouse
The data volume of tables of data all carries out comprehensive monitoring, can more effectively prevention data missing omit, being capable of effective guarantee number
According to the stability in warehouse.
It is the structure diagram of the monitoring system of the data warehouse of one embodiment of the invention referring to Fig. 1, including：Increment
Prediction module obtains any data table in data warehouse, and moving average model prediction data table is integrated pre- according to autoregression
The data increment in the fixed time period of day is surveyed, data increment is the increase item number of data in tables of data；Increment comparing module is used
In calculating real data increment of the prediction day in fixed time period, real data increment is compared with the data increment predicted
Right, whether monitoring data table carries out early warning.
Specifically, in embodiments of the present invention, incremental forecasting module predicts the fixed time period predicted in day for acquisition
The data increment of interior tables of data, and real data of the increment contrast module on the day of obtaining prediction day in fixed time period increases
Amount, and real data increment is compared with the data increment predicted, judge whether deviation between the two belongs to normal model
It encloses, if it is not, then carrying out early warning to the tables of data, effective prevention data missing is come with this and is omitted.It should be noted that data
The implementation steps above method embodiment of the specific monitoring and early warning of any data table progress has elaborated in warehouse, herein not
It is again to repeat more.
The embodiment of the present invention is by the present invention by by the actual increment of the data gathered from data warehouse and autoregression
The increment of integration moving average model prediction is compared, and early warning is carried out in actual increment and excessive prediction increment deviation, and
It realizes and comprehensive monitoring is carried out to the data volume of each tables of data in data warehouse, it can be with effective guarantee data warehouse
Stability.
Finally, the present processes are only preferable embodiment, are not intended to limit the scope of the present invention.It is all
Within the spirit and principles in the present invention, any modifications, equivalent replacements and improvements are made should be included in the protection of the present invention
Within the scope of.
Claims (10)
1. a kind of monitoring method of data warehouse, which is characterized in that including：
S1 obtains any data table in data warehouse, and integrating moving average model according to autoregression predicts that the tables of data exists
Predict the data increment in the fixed time period of day, the data increment is the increase item number of data in tables of data；
S2 calculates real data increment of the prediction day in fixed time period, by the real data increment and prediction
The data increment is compared, and monitors whether the tables of data carries out early warning.
2. the monitoring method of data warehouse according to claim 1, which is characterized in that include before step S1：
Database name and data table name in the data warehouse, search any data table in the data warehouse.
3. the monitoring method of data warehouse according to claim 1, the autoregression integration moving average model is included certainly
Regression model, moving average model and autoregressive moving-average model, which is characterized in that the integration of autoregression described in step S1 is slided
Dynamic averaging model is established by following steps：
The data increment of the fixed time period in described prediction continuous several weeks a few days ago is formed into time series, by the time sequence
Column processing is stationary time series；
According to the partial autocorrelation function and auto-correlation function of the stationary time series, establish corresponding with the stationary time series
Autoregression integration moving average model.
4. the monitoring method of data warehouse according to claim 3, which is characterized in that described by the time Series Processing
Include for the specific steps of stationary time series：
It is nonstationary time series according to the time series, first difference is carried out to each data increment in the time series
Partite transport is calculated, and obtains new time series；
Judge whether the new time series is stationary time series, if it is not, previous step is then returned to, until the new time
Sequence is stationary time series.
5. the monitoring method of data warehouse according to claim 3, which is characterized in that described according to the stationary time sequence
The partial autocorrelation function and auto-correlation function of row establish autoregression integration sliding average mould corresponding with the stationary time series
Type specifically includes：
If the partial autocorrelation function is with truncation and auto-correlation function is with hangover property, establish and the stationary time series
Corresponding autoregression model；
If partial autocorrelation function is with hangover property and auto-correlation function is with truncation, establish corresponding with the stationary time series
Moving average model；
If partial autocorrelation function and auto-correlation function are respectively provided with hangover, autoregression corresponding with the stationary time series is established
Moving average model；
Wherein, the determination methods of the hangover property are function by exponential damping or into sine wave, the judgement of the truncation
Method is that functional value is zero.
6. the monitoring method of data warehouse according to claim 1, which is characterized in that fixed time period described in step S1
It is obtained by following steps：
Multiple special time periods were chosen in one week, gather the tables of data within continuous several weeks in each special time period
Data increment, calculate the variance of the data increment in each special time period；
Compare the variance of data increment in all special time periods, using the special time period of variance minimum as the fixation
Period.
7. the monitoring method of data warehouse according to claim 1, which is characterized in that real data described in step S2 increases
Amount is obtained by following steps：
Record the write time in each tables of data per data in the data warehouse；By described in the calculating of said write time
The data increment of tables of data in fixed time period.
8. the monitoring method of data warehouse according to claim 7, which is characterized in that described to calculate each prediction day
Real data increment in fixed time period specifically includes：
By the said write time, the primary data total amount of the tables of data when fixed time period starts, Yi Jisuo are gathered
State the end total amount of data of the tables of data at the end of fixed time period；
The end total amount of data is subtracted into the primary data total amount, obtains the number of the tables of data in the fixed time period
According to increment.
9. the monitoring method of data warehouse according to claim 1, which is characterized in that by the reality described in step S2
Data increment is compared with the data increment predicted, monitors the specific steps the bag whether tables of data carries out early warning
It includes：
The ratio of the real data increment and the prediction data increment is subtracted 1 as deviation, if the deviation is exhausted
10% is more than to value, early warning is carried out to the tables of data.
10. a kind of monitoring system of data warehouse, which is characterized in that including：
Incremental forecasting module obtains any data table in data warehouse, and moving average model prediction institute is integrated according to autoregression
Data increment of the tables of data in the fixed time period of prediction day is stated, the data increment is the increase item of data in tables of data
Number；
Increment comparing module, for calculating real data increment of the prediction day in fixed time period, by the actual number
It is compared according to increment with the data increment predicted, monitors whether the tables of data carries out early warning.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711230833.7A CN108090138A (en) | 2017-11-29 | 2017-11-29 | The monitoring method and system of a kind of data warehouse |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711230833.7A CN108090138A (en) | 2017-11-29 | 2017-11-29 | The monitoring method and system of a kind of data warehouse |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108090138A true CN108090138A (en) | 2018-05-29 |
Family
ID=62173442
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711230833.7A Pending CN108090138A (en) | 2017-11-29 | 2017-11-29 | The monitoring method and system of a kind of data warehouse |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108090138A (en) |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109543993A (en) * | 2018-11-20 | 2019-03-29 | 青海黄河上游水电开发有限责任公司光伏产业技术分公司 | Analyze method, computer storage medium and the computer equipment of photovoltaic plant |
| CN109684300A (en) * | 2018-11-20 | 2019-04-26 | 成都四方伟业软件股份有限公司 | One kind being based on visual big data warehouse design method and system |
| CN110083638A (en) * | 2019-04-24 | 2019-08-02 | 广东联合电子服务股份有限公司 | A kind of regular base construction method of delay and data retention analysis method |
| CN110134680A (en) * | 2019-04-04 | 2019-08-16 | 平安科技（深圳）有限公司 | Space monitoring method, device, computer equipment and storage medium |
| CN110210959A (en) * | 2019-06-10 | 2019-09-06 | 广发证券股份有限公司 | Analysis method, device and the storage medium of financial data |
| CN110851325A (en) * | 2019-11-08 | 2020-02-28 | 深圳市彬讯科技有限公司 | Method, device and equipment for monitoring data warehouse based on Hive table |
| CN110888775A (en) * | 2019-11-08 | 2020-03-17 | 深圳市彬讯科技有限公司 | Method, device and equipment for monitoring data warehouse by data balance |
| CN113407422A (en) * | 2021-08-20 | 2021-09-17 | 太平金融科技服务(上海)有限公司深圳分公司 | Data abnormity alarm processing method and device, computer equipment and storage medium |
| CN113821409A (en) * | 2021-09-23 | 2021-12-21 | 中国建设银行股份有限公司 | Method, device, storage medium and equipment for monitoring data transmission |
| CN114322634A (en) * | 2021-12-29 | 2022-04-12 | 博锐尚格科技股份有限公司 | Data screening method and device for refrigerating system strategy model |
| CN113821409B (en) * | 2021-09-23 | 2024-06-04 | 中国建设银行股份有限公司 | Method, device, storage medium and equipment for monitoring data transmission |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104301895A (en) * | 2014-09-28 | 2015-01-21 | 北京邮电大学 | Double-layer trigger intrusion detection method based on flow prediction |
| CN104581749A (en) * | 2013-10-11 | 2015-04-29 | 北京亿阳信通科技有限公司 | A method and device for predicting service amount of data service of mobile network |
| CN105095056A (en) * | 2015-08-14 | 2015-11-25 | 焦点科技股份有限公司 | Method for monitoring data in data warehouse |
| CN106157163A (en) * | 2016-07-27 | 2016-11-23 | 河南工业大学 | A kind of grain yield short term prediction method and device |
| CN106844180A (en) * | 2017-02-07 | 2017-06-13 | 山东浪潮云服务信息科技有限公司 | A kind of monitoring and controlling forecast method of OpenStack platforms computing resource |
-
2017
- 2017-11-29 CN CN201711230833.7A patent/CN108090138A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104581749A (en) * | 2013-10-11 | 2015-04-29 | 北京亿阳信通科技有限公司 | A method and device for predicting service amount of data service of mobile network |
| CN104301895A (en) * | 2014-09-28 | 2015-01-21 | 北京邮电大学 | Double-layer trigger intrusion detection method based on flow prediction |
| CN105095056A (en) * | 2015-08-14 | 2015-11-25 | 焦点科技股份有限公司 | Method for monitoring data in data warehouse |
| CN106157163A (en) * | 2016-07-27 | 2016-11-23 | 河南工业大学 | A kind of grain yield short term prediction method and device |
| CN106844180A (en) * | 2017-02-07 | 2017-06-13 | 山东浪潮云服务信息科技有限公司 | A kind of monitoring and controlling forecast method of OpenStack platforms computing resource |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109543993A (en) * | 2018-11-20 | 2019-03-29 | 青海黄河上游水电开发有限责任公司光伏产业技术分公司 | Analyze method, computer storage medium and the computer equipment of photovoltaic plant |
| CN109684300A (en) * | 2018-11-20 | 2019-04-26 | 成都四方伟业软件股份有限公司 | One kind being based on visual big data warehouse design method and system |
| CN109543993B (en) * | 2018-11-20 | 2023-04-18 | 青海黄河上游水电开发有限责任公司光伏产业技术分公司 | Method for analyzing photovoltaic power station, computer storage medium and computer device |
| CN110134680A (en) * | 2019-04-04 | 2019-08-16 | 平安科技（深圳）有限公司 | Space monitoring method, device, computer equipment and storage medium |
| CN110083638A (en) * | 2019-04-24 | 2019-08-02 | 广东联合电子服务股份有限公司 | A kind of regular base construction method of delay and data retention analysis method |
| CN110210959A (en) * | 2019-06-10 | 2019-09-06 | 广发证券股份有限公司 | Analysis method, device and the storage medium of financial data |
| CN110888775A (en) * | 2019-11-08 | 2020-03-17 | 深圳市彬讯科技有限公司 | Method, device and equipment for monitoring data warehouse by data balance |
| CN110851325A (en) * | 2019-11-08 | 2020-02-28 | 深圳市彬讯科技有限公司 | Method, device and equipment for monitoring data warehouse based on Hive table |
| CN110851325B (en) * | 2019-11-08 | 2024-03-15 | 土巴兔集团股份有限公司 | Method, device and equipment for monitoring data warehouse based on Hive table |
| CN110888775B (en) * | 2019-11-08 | 2024-04-09 | 土巴兔集团股份有限公司 | Method, device and equipment for monitoring data warehouse by utilizing data balance |
| CN113407422A (en) * | 2021-08-20 | 2021-09-17 | 太平金融科技服务(上海)有限公司深圳分公司 | Data abnormity alarm processing method and device, computer equipment and storage medium |
| CN113407422B (en) * | 2021-08-20 | 2021-11-09 | 太平金融科技服务(上海)有限公司深圳分公司 | Data abnormity alarm processing method and device, computer equipment and storage medium |
| CN113821409A (en) * | 2021-09-23 | 2021-12-21 | 中国建设银行股份有限公司 | Method, device, storage medium and equipment for monitoring data transmission |
| CN113821409B (en) * | 2021-09-23 | 2024-06-04 | 中国建设银行股份有限公司 | Method, device, storage medium and equipment for monitoring data transmission |
| CN114322634A (en) * | 2021-12-29 | 2022-04-12 | 博锐尚格科技股份有限公司 | Data screening method and device for refrigerating system strategy model |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108090138A (en) | The monitoring method and system of a kind of data warehouse | |
| US7788127B1 (en) | Forecast model quality index for computer storage capacity planning | |
| Segnon et al. | Modeling and forecasting the volatility of carbon dioxide emission allowance prices: A review and comparison of modern volatility models | |
| Raggi et al. | Long memory and nonlinearities in realized volatility: a Markov switching approach | |
| Marcellino | A linear benchmark for forecasting GDP growth and inflation? | |
| Mancini et al. | Spot volatility estimation using delta sequences | |
| Berger et al. | Nowcasting the output gap | |
| US20100010869A1 (en) | Demand curve analysis method for predicting forecast error | |
| CN106649832B (en) | Estimation method and device based on missing data | |
| US20150235133A1 (en) | Data concentration prediction device, data concentration prediction method, and recording medium recording program thereof | |
| Marczak et al. | Outlier detection in structural time series models: The indicator saturation approach | |
| US20190303863A1 (en) | Database modification for improved on-shelf availability determination | |
| Bergman et al. | A Bayesian approach to demand forecasting for new equipment programs | |
| CN110909306B (en) | Business abnormality detection method and device, electronic equipment and storage equipment | |
| CN113347057B (en) | Abnormal data detection method and device, electronic equipment and storage medium | |
| Clements et al. | Data revisions and real-time forecasting | |
| CN112150205A (en) | Price prediction method and device and electronic equipment | |
| CN112070284A (en) | Screening method, device, equipment and storage medium for component prediction | |
| Bisi et al. | Dynamic learning, pricing, and ordering by a censored newsvendor | |
| Huseby et al. | Discrete event simulation methods applied to advanced importance measures of repairable components in multistate network flow systems | |
| Bjørnland et al. | Forecasting inflation with an uncertain output gap | |
| CN107590244B (en) | Method and device for identifying offline activity scene of mobile equipment | |
| US7783509B1 (en) | Determining that a change has occured in response to detecting a burst of activity | |
| CN110992189A (en) | Resource data estimation method, resource data estimation device, computer equipment and storage medium | |
| Thury et al. | Forecasting industrial production using structural time series models |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180529 |