CN117891853A - Data characteristic extraction method and system of time sequence database and electronic equipment - Google Patents
Data characteristic extraction method and system of time sequence database and electronic equipment Download PDFInfo
- Publication number
- CN117891853A CN117891853A CN202410058101.8A CN202410058101A CN117891853A CN 117891853 A CN117891853 A CN 117891853A CN 202410058101 A CN202410058101 A CN 202410058101A CN 117891853 A CN117891853 A CN 117891853A
- Authority
- CN
- China
- Prior art keywords
- statistical analysis
- data
- time
- time sequence
- sequence data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000605 extraction Methods 0.000 title claims abstract description 13
- 230000006870 function Effects 0.000 claims abstract description 117
- 238000007619 statistical method Methods 0.000 claims abstract description 112
- 238000000034 method Methods 0.000 claims abstract description 34
- 238000004364 calculation method Methods 0.000 claims abstract description 29
- 230000008859 change Effects 0.000 claims description 21
- 238000004458 analytical method Methods 0.000 claims description 19
- 238000012417 linear regression Methods 0.000 claims description 12
- 238000010183 spectrum analysis Methods 0.000 claims description 8
- 238000004590 computer program Methods 0.000 claims description 6
- 238000003860 storage Methods 0.000 claims description 3
- 230000003139 buffering effect Effects 0.000 claims description 2
- 230000001419 dependent effect Effects 0.000 description 10
- 230000003595 spectral effect Effects 0.000 description 10
- 238000012545 processing Methods 0.000 description 8
- 230000008569 process Effects 0.000 description 7
- 238000007405 data analysis Methods 0.000 description 3
- 238000013075 data extraction Methods 0.000 description 3
- 235000019580 granularity Nutrition 0.000 description 3
- 230000010355 oscillation Effects 0.000 description 3
- 238000005070 sampling Methods 0.000 description 3
- 238000004220 aggregation Methods 0.000 description 2
- 230000002776 aggregation Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000001228 spectrum Methods 0.000 description 2
- 230000001960 triggered effect Effects 0.000 description 2
- 238000009825 accumulation Methods 0.000 description 1
- 230000001186 cumulative effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 238000009776 industrial production Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000000611 regression analysis Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000012163 sequencing technique Methods 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2462—Approximate or statistical queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2308—Concurrency control
- G06F16/2315—Optimistic concurrency control
- G06F16/2322—Optimistic concurrency control using timestamps
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2474—Sequence data queries, e.g. querying versioned data
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Probability & Statistics with Applications (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Complex Calculations (AREA)
Abstract
The invention discloses a data characteristic extraction method, a system and electronic equipment of a time sequence database, and relates to the technical field of data characteristic extraction, wherein the method comprises the following steps: acquiring time sequence data in real time, and caching each time sequence data in a memory queue; using the statistical analysis function of the statistical analysis callback function to perform statistical calculation on the time sequence data in the memory queue according to five minutes to obtain a statistical analysis result; when new time sequence data is acquired, performing history compensation operation; the history rebate operation includes: when new time sequence data is acquired, triggering a corresponding event, informing a related statistical analysis function to perform a time-compensating operation, checking whether the time sequence data is updated or not in a fixed time interval, informing the related statistical analysis function to perform the time-compensating operation, and thus completing the alignment of time stamps of the time sequence data according to the fixed time interval.
Description
Technical Field
The present invention relates to the field of data characteristic extraction technologies, and in particular, to a method, a system, and an electronic device for extracting data characteristics of a time-series database.
Background
With the popularity of the internet of things (Internet ofThings, ioT) and industry 4.0, mass data is produced by various industries, with time stamping being a major feature. A real-time database is a software product that is capable of handling fast changing data and transactions with time constraints. However, the conventional industrial real-time database can only provide simple historical data curve query and statistical query, and cannot meet the functions of multidimensional aggregation calculation, data analysis and the like.
Disclosure of Invention
The invention aims to provide a data characteristic extraction method, a system and electronic equipment of a time sequence database, which are used for aligning time stamps of data according to fixed time intervals.
In order to achieve the above object, the present invention provides the following solutions:
A method for extracting data characteristics of a time series database, comprising:
acquiring time sequence data in real time, and caching each time sequence data in a memory queue;
Using a statistical analysis function of a statistical analysis callback function to perform statistical calculation on the time sequence data in the memory queue according to five minutes to obtain a statistical analysis result; when new time sequence data is acquired, performing history compensation operation;
The history back-filling operation includes: when new time sequence data is acquired, triggering a corresponding event, informing a related statistical analysis function to perform a compensation operation, checking whether the time sequence data is updated or not in a fixed time interval, and informing the related statistical analysis function to perform the compensation operation.
Optionally, after obtaining the statistical analysis result, the method further includes:
and storing the statistical analysis result.
Optionally, the statistical analysis function includes: basic statistics function, linear regression analysis function, spectrum analysis function and custom analysis function.
Optionally, the basic statistical function includes: time weighted average, moving average, rate of change, statistics, maximum and minimum, total, variance, standard deviation, and variation.
Optionally, storing the statistical analysis result includes:
pre-counting the statistical analysis results of five minutes, and storing the statistical analysis results into a minute clock;
Pre-counting the statistical analysis result of each hour and storing the statistical analysis result into an hour table;
The statistical analysis results of each day are pre-counted and stored in a day table.
Optionally, when new time sequence data is acquired, triggering a corresponding event, and notifying a related statistical analysis function to perform a refill operation, including:
registering a data write event listener in the system;
When new time sequence data is acquired, the data is written into event notification data to be written into an event monitor in a mode of statistically analyzing callback functions;
After receiving the data writing event, the data writing event monitor calls back the related statistical analysis function to perform statistical calculation on the new time sequence data.
Optionally, checking whether the time series data is updated in a fixed time interval, and notifying the relevant statistical analysis function to perform the interpolation operation, including:
Starting a timing task thread;
triggering the timing task thread in a set time interval, and comparing the ID of the tag stored by the system with the ID of the tag stored by the timing task thread;
If the IDs are consistent, comparing the last stored time of the labels stored by the system with the last stored time of the labels stored by the timed task thread;
if the last stored time of the labels stored in the system is later than the last stored time of the labels stored in the timed task thread, judging that the time sequence data is updated, and informing a related statistical analysis function to perform statistical calculation on the new time sequence data.
A data characteristic extraction system of a time series database, comprising:
The time sequence data acquisition module is used for acquiring time sequence data in real time and buffering each time sequence data in the memory queue;
The statistical analysis module is used for carrying out statistical calculation on the time sequence data in the memory queue according to five minutes by utilizing the statistical analysis function of the statistical analysis callback function to obtain a statistical analysis result; when new time sequence data is acquired, performing history compensation operation;
The history back-filling operation includes: when new time sequence data is acquired, triggering a corresponding event, informing a related statistical analysis function to perform a compensation operation, checking whether the time sequence data is updated or not in a fixed time interval, and informing the related statistical analysis function to perform the compensation operation.
An electronic device comprising a memory for storing a computer program and a processor that runs the computer program to cause the electronic device to perform the method of extracting data characteristics of a time series database described above.
Optionally, the memory is a readable storage medium.
According to the specific embodiment provided by the invention, the invention discloses the following technical effects:
The invention discloses a data characteristic extraction method, a system and electronic equipment of a time sequence database, which are characterized in that firstly, time sequence data are acquired in real time, and each time sequence data are cached in a memory queue; then, using the statistical analysis function of the statistical analysis callback function to perform statistical calculation on the time sequence data in the memory queue according to five minutes to obtain a statistical analysis result; when new time sequence data is acquired, performing history compensation operation; the history rebate operation includes: when new time sequence data is acquired, triggering a corresponding event, informing a related statistical analysis function to perform a re-filling operation, checking whether the time sequence data is updated or not in a fixed time interval, and informing the related statistical analysis function to perform the re-filling operation, thereby realizing the alignment of the time stamps of the data according to the fixed time interval.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a flow chart of a method for extracting data characteristics of a time-series database according to embodiment 1 of the present invention;
Fig. 2 is a schematic diagram of a data extraction technique.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The invention aims to provide a data characteristic extraction method, a system and electronic equipment of a time sequence database, aiming at aligning time stamps of data according to fixed time intervals.
In order that the above-recited objects, features and advantages of the present invention will become more readily apparent, a more particular description of the invention will be rendered by reference to the appended drawings and appended detailed description.
Example 1
Fig. 1 is a flowchart illustrating a method for extracting data characteristics of a time-series database according to embodiment 1 of the present invention. As shown in fig. 1, the data characteristic extraction method of the time-series database in the present embodiment includes:
Step 101: and acquiring time sequence data in real time, and caching each time sequence data in a memory queue.
Step 102: and carrying out statistical calculation on the time sequence data in the memory queue according to five minutes by utilizing a statistical analysis function of the statistical analysis callback function to obtain a statistical analysis result.
When new time sequence data is acquired, history compensation operation is carried out.
The history rebate operation includes: when new time sequence data is acquired, triggering a corresponding event, informing a related statistical analysis function to perform a compensation operation, checking whether the time sequence data is updated or not in a fixed time interval, and informing the related statistical analysis function to perform the compensation operation.
As an alternative embodiment, after obtaining the statistical analysis result, the method further includes:
and storing the statistical analysis result.
As an alternative embodiment, the statistical analysis function includes: basic statistics function, linear regression analysis function, spectrum analysis function and custom analysis function.
As an alternative embodiment, the basic statistical functions include: time weighted average, moving average, rate of change, statistics, maximum and minimum, total, variance, standard deviation, and variation.
As an alternative embodiment, storing the statistical analysis result includes:
the statistical analysis results of five minutes were pre-counted and stored in a minute clock.
The statistical analysis results of each hour are pre-counted and stored in an hour table.
The statistical analysis results of each day are pre-counted and stored in a day table.
As an optional implementation manner, when new time series data is acquired, a corresponding event is triggered, and the relevant statistical analysis function is notified to perform a refill operation, which includes:
a data write event listener is registered in the system.
When new time sequence data is acquired, the data is written into the event notification data by means of a statistical analysis callback function and is written into the event monitor.
After receiving the data writing event, the data writing event monitor calls back the related statistical analysis function to perform statistical calculation on the new time sequence data.
As an alternative embodiment, checking whether the time series data is updated in a fixed time interval, and notifying the relevant statistical analysis function to perform the interpolation operation includes:
a timed task thread is started.
Triggering the timing task thread in a set time interval, and comparing the ID of the tag stored by the system with the ID of the tag stored by the timing task thread.
If the IDs are consistent, comparing the last saved time of the tags saved by the system with the last saved time of the tags saved by the timed task thread.
If the last stored time of the labels stored in the system is later than the last stored time of the labels stored in the timed task thread, judging that the time sequence data is updated, and informing a related statistical analysis function to perform statistical calculation on the new time sequence data.
Further, in order to implement the method in the embodiment, a data extraction technology is also provided, as shown in fig. 2, where the data extraction technology specifically includes the following functions:
1. data ordering and archiving.
Firstly, sequencing time sequence data of the same sampling point collected by a system according to a time sequence so as to ensure time continuity of the data. Then, the time sequence data is cached in a memory queue, and the time sequence data is archived according to the minimum time interval of 5 minutes. The data of the 5 minute time interval is the smallest statistical and calculation interval, so that basic data can be provided for the subsequent statistical functions.
2. And (5) calculating statistics.
The scalable statistical analysis callback function can implement the data statistical analysis of each 5-minute time-series data segment by:
1. defining a callback function interface: first, a callback function interface needs to be defined, and the callback function interface comprises a method for carrying out statistical analysis on the data segments. Different parameters and return value types can be defined as needed to meet various statistical requirements. The definition of the callback function interface is implemented using a function pointer.
2. The basic statistical function is realized: various common statistical functions, such as time weighted average, moving average, rate of change, statistics, maximum and minimum, etc., are implemented in the statistical analysis callback function. These functions may be implemented by performing iterative computations over the data segments.
(1) Time weighted average: two variables are defined in the statistical analysis callback function, one is used for accumulating the weighted sum of the data, the other is used for accumulating the time weight, and the data is weighted and accumulated according to the time weight during each iteration.
The calculation formula of the time weighted average is: weighted average= (x1×w1+x2×w2+x3×w3+ + xn×wn)/(w1+w2+w3+ + wn).
Wherein X1, X2, X3,..and Xn are values in adjacent time periods, and w1, w2, w3,..and wn are weights for the corresponding time periods.
(2) Moving average: a queue or sliding window is used in the statistical analysis callback function to hold the data for the last period of time, the queue or sliding window is updated at each iteration, and the average is calculated.
The calculation formula of the exponential moving average (Exponential MovingAverage, EMA) is: EMA = (x× (2/(n+1))) + (ema_previous× (1- (2/(n+1)))).
Wherein X is the value of the current time period, n is the length of the time period, and EMA_previous is the exponential moving average of the previous time period.
(3) Rate of change: and storing the data of the last iteration in the statistical analysis callback function, calculating the difference value between the current data and the last data in each iteration, and calculating the change rate.
The calculation formula of the change rate is as follows: change rate= ((new value-old value)/old value) ×100.
The rate of change can be expressed as a percentage by dividing the amount of change by the old value and multiplying by 100. This formula is applicable to calculate the rate of change between any two values.
(4) Statistics number, maximum and minimum: and defining variables in the statistical analysis callback function to save a statistical result, and updating the number and the maximum and minimum values when each iteration is performed.
Expanding advanced statistics functions: in addition to basic statistical functions, higher-level statistical functions such as aggregate, variance, standard deviation, variance, etc. can be extended. Corresponding calculation logic can be added to the statistical analysis callback function to realize the functions according to the requirements.
(5) Totaling: a variable is defined in the statistical analysis callback function, and data is accumulated into the variable during each iteration to obtain an aggregate result.
The total calculation formula is: total = Σdi.
Where di is the value of the ith data point, Σ represents summing all data points.
(6) Variance and standard deviation: two variables are defined in the statistical analysis callback function, one is used for accumulating square difference values, the other is used for accumulating data values, the square difference values are calculated and accumulated into phase dependent variable in each iteration, and finally the variance and standard deviation are calculated according to accumulation results.
The variance is calculated as:
Variance = (Σ (di- μ) 2)/m
Where μ is the average of all data points and m is the number of data points.
The calculation formula of the standard deviation is as follows: standard deviation= (variance) 1/2.
(7) The change is: and storing the data of the last iteration in the statistical analysis callback function, calculating the difference value between the current data and the last data in each iteration, and judging the change direction according to the positive and negative conditions of the difference value.
The calculation formula of the change is as follows: change = new value-old value.
This formula can be used to calculate the change between two points in time, or between any two values.
3. Predefined algorithms and models can be used for deeper data analysis and modeling, including linear regression analysis and spectral analysis. These algorithms and models may be incorporated into the steps described above to achieve a more comprehensive data statistics analysis function.
(1) Linear regression analysis: regression analysis of the data segment may be implemented in the statistical analysis callback function. This can help the user discover potential associations between variables and make predictions and decisions.
The linear regression model is a basic statistical learning method, and predicts the value of a dependent variable by establishing a linear equation for the relationship between the independent variable and the dependent variable. Specifically, assuming a set of independent variables x and corresponding dependent variables y, the general expression of the linear regression model is:
y＝a+bx+e。
Wherein a and b are coefficients of a linear regression model, respectively, for describing a linear relationship of y to x, a being an intercept; b is a regression coefficient; e is an error term used to represent noise of samples that have not been interpreted by the linear regression model.
The goal of the linear regression model is to find the best parameters so that the total error of the sample is minimized, which can be achieved by the least squares method. The core of the least squares method is the definition and optimization of an objective function, which can be defined as the sum of squares of the distances between the sample points and the fitted straight line.
The specific process of fitting the trendline of the data segment using the linear regression model is:
1) And determining the data segment to be fitted.
2) The independent variable and the dependent variable are placed in a vector, respectively.
3) The mean of the independent and dependent variables is calculated.
4) The independent and dependent standard deviations are calculated.
5) Covariance of the independent variables and dependent variables is calculated.
6) The regression coefficient b and the intercept a are calculated as follows:
b＝Cov(x，y)/Var(x)。
a＝y_mean-b×x_mean。
wherein Cov (·) is covariance; var (·) is variance; y_mean is the mean of y; x_mean is the mean of x.
7) And obtaining a fitting curve according to the regression coefficient and the intercept obtained by calculation.
8) The correlation coefficient r is calculated to evaluate the strength and correlation of the trend, and the specific calculation mode is as follows:
r＝Cov(x，y)/(S(x)×S(y))
Wherein S (·) is the standard deviation.
A linear regression model may be used to interpret the linear relationship between the independent and dependent variables, but if the relationship between the independent and dependent variables is nonlinear, then other types of regression models may need to be used.
(2) Spectral analysis: the spectral analysis of the data segments may be implemented in a statistical analysis callback function. Fourier transforms are used to transform time domain data into frequency domain data to analyze the frequency content and periodic characteristics of the data. The spectral characteristics of the data are revealed by calculating the power spectral density and the spectrogram so as to help users understand the periodicity and oscillation characteristics of the data.
The specific process of spectrum analysis of the data segment is as follows:
1) A data segment to be analyzed is determined.
2) The data segment to be analyzed is discretized into a set of equally spaced sampling points according to the sampling rate.
3) The time domain data is converted into frequency domain data using fourier transform.
4) And calculating the amplitude and the phase of the frequency domain data obtained by the Fourier transform.
5) And calculating the power spectral density and the spectrogram according to the amplitude and the phase.
6) The power spectral density and the spectrogram are visualized to reveal spectral features of the data.
Fourier transform is a method of converting a time domain signal into a frequency domain signal, converting a signal varying in the time domain into an amplitude spectrum in the frequency domain. The principle of the Fourier transform is to transform the information in the time domain into the energy distribution in the frequency domain, and the amplitude and the phase of the sinusoidal curve are used for representing the characteristics of the data in the frequency domain, so that the data periodicity and the oscillation characteristics can be revealed.
The specific process of calculating the power spectral density and the spectrogram is as follows:
1) The amplitude and phase are calculated.
2) And squaring the amplitude to obtain the power spectrum density.
3) The power spectral density is converted into decibel units in order to better exhibit the power differences.
4) The power spectral density is plotted as a spectrogram, with the abscissa being frequency and the ordinate being power spectral density, so as to better characterize the frequency components.
After the power spectral density is calculated and the spectrogram is drawn, the periodicity and oscillation characteristics of the data can be observed, and the characteristics of the data can be extracted through further analysis and used in the application of signal processing and spectral analysis.
4. The user self-defines a calculation formula: to meet more complex statistical requirements, a user may be allowed to customize the calculation formula to calculate data for the relevant acquisition point. For example, a user may calculate the flow of a certain pipe by flow rate and time. The user-defined formulas may be parsed in a statistical analysis callback function and the relevant data calculated from the formulas.
Through the steps, the extensible statistical analysis callback function can realize data statistical analysis on each data segment of 5 minutes and meet various common and advanced statistical requirements. Such a design may provide flexibility and scalability, making the statistical analysis more powerful and adaptable to a variety of scenarios.
3. And storing statistical analysis results.
In order to improve the data query efficiency, the statistic analysis and statistic analysis callback function performs pre-statistics on the five-minute data of each measuring point, stores the pre-statistics once every hour into an hour table, performs pre-statistics once every day, and stores the results into a day table. In this way, the data is pre-counted according to different time granularities to meet the query requirements of different granularities. Thus, the data query speed can be greatly increased, and the access to the original data is reduced.
4. And (5) historical supplement notification.
In order to maintain the accuracy and the real-time performance of the statistical analysis data, the function of historical feedback notification is realized. When new time sequence data is written, the system can detect and timely call back related analysis and statistics functions so as to keep consistency of data statistics and analysis results. The realization of the history feedback notification function can adopt the following technical means and specific processes:
1. Listening for data write events: the data write event may be listened to by setting a listener in the system. When new time sequence data is written, the system triggers corresponding events and notifies the relevant statistical analysis function to carry out the compensation operation. The specific process is as follows:
(1) Registration data write event listener: registering a data writing event monitor in the system, which is realized by using a self-defined statistical analysis callback function and is used for monitoring the data writing event.
(2) Listening for data write events: after the data writing operation is completed, notifying the monitor by means of statistically analyzing the callback function. After the data is written, the system can automatically call a statistical analysis callback function, and the written data is transferred to a monitor for processing.
(3) Callback statistics analysis function: after receiving the data writing event, the monitor will call back the related statistical analysis function, such as time weighted average, moving average, change rate, statistical quantity, maximum and minimum values and total, variance, standard deviation, change, etc., and transfer the newly written data to the statistical analysis function for corresponding processing and updating.
2. Periodically checking the data: in addition to listening for data write events, a history rebate notification function is implemented in a manner that periodically checks for data. A timed task thread may be provided to check whether the data is updated at regular intervals and to notify the relevant statistical analysis function to perform the refill operation. The specific process is as follows:
(1) Setting a timed task thread: a thread is started in the system, which is executed every 5 minutes to periodically check whether the data is updated.
(2) Checking data updating: the timing task thread is triggered within a set time interval, the ID of the tag stored in the system and the last stored time are compared with the ID of the tag stored in the timing task thread and the last stored time, if the last stored time of the system is later than the last stored time of the timing task thread, the data is considered to be updated, and the latest data information is acquired.
(3) Callback statistics analysis function: if the data is updated, the system informs the relevant statistical analysis function to perform the back-filling operation, and transmits the new data to the statistical analysis function for processing and updating.
Through the technical means and the specific process, the history feedback notification function can be realized. Therefore, the data statistics analysis result can be kept synchronous with the original data, the change of the data can be reflected timely, and the accuracy and the instantaneity of the data are maintained. Meanwhile, the data query efficiency can be improved, and a reliable basis is provided for data analysis and decision making in industrial production.
Example 2
The data characteristic extraction system of the time series database in the present embodiment includes:
the time sequence data acquisition module is used for acquiring time sequence data in real time and caching each time sequence data in the memory queue.
The statistical analysis module is used for carrying out statistical calculation on the time sequence data in the memory queue according to five minutes by utilizing the statistical analysis function of the statistical analysis callback function to obtain a statistical analysis result; when new time sequence data is acquired, history compensation operation is carried out.
The history rebate operation includes: when new time sequence data is acquired, triggering a corresponding event, informing a related statistical analysis function to perform a compensation operation, checking whether the time sequence data is updated or not in a fixed time interval, and informing the related statistical analysis function to perform the compensation operation.
Example 3
An electronic device comprising a memory for storing a computer program and a processor that runs the computer program to cause the electronic device to perform the data characteristic extraction method of the time series database in embodiment 1.
As an alternative embodiment, the memory is a readable storage medium.
The invention has the following advantages:
(1) The query efficiency is improved: the data processing is carried out on the five-minute data of each measuring point to obtain results such as cumulative summation, time weighted average, moving average, change rate, statistical quantity, maximum and minimum value, total, variance, standard deviation, change and the like, and the results are stored in minute, hour and day meters, so that the data query efficiency can be greatly improved. When inquiring, only the result is needed to be obtained from the corresponding table, and complex calculation and aggregation operation on the original data are not needed.
(2) The calculated amount is reduced: the statistical analysis callback function aggregates the original data according to different time granularities, and merges a large amount of data into a small amount of statistical results. Thus, the calculation amount can be greatly reduced, and the processing capacity and the response speed of the system are improved.
(3) Maintaining data accuracy and real-time: the time sequence data updating and history feedback notification function can timely notify the statistical analysis callback function so as to update the pre-statistical data, and the accuracy and the instantaneity of the data are maintained. When new data is written, the system can detect and update related data in time, and consistency of data statistics analysis results is ensured.
In summary, the statistical analysis callback function has important advantages in time series data processing and analysis. By pre-counting the data, the query efficiency can be improved, the calculated amount can be reduced, the accuracy and the instantaneity of the data can be kept, and the system performance and the expandability can be improved. These advantages make it an important means of processing time series data.
In the present specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, and identical and similar parts between the embodiments are all enough to refer to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant points refer to the description of the method section.
The principles and embodiments of the present invention have been described herein with reference to specific examples, the description of which is intended only to assist in understanding the methods of the present invention and the core ideas thereof; also, it is within the scope of the present invention to be modified by those of ordinary skill in the art in light of the present teachings. In view of the foregoing, this description should not be construed as limiting the invention.
Claims (10)
1. A method for extracting data characteristics of a time series database, the method comprising:
acquiring time sequence data in real time, and caching each time sequence data in a memory queue;
Using a statistical analysis function of a statistical analysis callback function to perform statistical calculation on the time sequence data in the memory queue according to five minutes to obtain a statistical analysis result; when new time sequence data is acquired, performing history compensation operation;
The history back-filling operation includes: when new time sequence data is acquired, triggering a corresponding event, informing a related statistical analysis function to perform a compensation operation, checking whether the time sequence data is updated or not in a fixed time interval, and informing the related statistical analysis function to perform the compensation operation.
2. The method for extracting data characteristics from a time series database according to claim 1, further comprising, after obtaining the statistical analysis result:
and storing the statistical analysis result.
3. The method for extracting data characteristics from a time series database according to claim 1, wherein the statistical analysis function includes: basic statistics function, linear regression analysis function, spectrum analysis function and custom analysis function.
4. A method of extracting data characteristics from a time series database according to claim 3, wherein the basic statistical function includes: time weighted average, moving average, rate of change, statistics, maximum and minimum, total, variance, standard deviation, and variation.
5. The method for extracting data characteristics from a time series database according to claim 2, wherein storing the statistical analysis result comprises:
pre-counting the statistical analysis results of five minutes, and storing the statistical analysis results into a minute clock;
Pre-counting the statistical analysis result of each hour and storing the statistical analysis result into an hour table;
The statistical analysis results of each day are pre-counted and stored in a day table.
6. The method for extracting data characteristics from a time series database according to claim 1, wherein when new time series data is acquired, triggering a corresponding event and notifying a relevant statistical analysis function to perform a back-filling operation comprises:
registering a data write event listener in the system;
When new time sequence data is acquired, the data is written into event notification data to be written into an event monitor in a mode of statistically analyzing callback functions;
After receiving the data writing event, the data writing event monitor calls back the related statistical analysis function to perform statistical calculation on the new time sequence data.
7. The method of claim 1, wherein checking whether the time series data is updated within a fixed time interval and notifying the relevant statistical analysis function to perform the back-filling operation comprises:
Starting a timing task thread;
triggering the timing task thread in a set time interval, and comparing the ID of the tag stored by the system with the ID of the tag stored by the timing task thread;
If the IDs are consistent, comparing the last stored time of the labels stored by the system with the last stored time of the labels stored by the timed task thread;
if the last stored time of the labels stored in the system is later than the last stored time of the labels stored in the timed task thread, judging that the time sequence data is updated, and informing a related statistical analysis function to perform statistical calculation on the new time sequence data.
8. A data characteristic extraction system of a time series database, the system comprising:
The time sequence data acquisition module is used for acquiring time sequence data in real time and buffering each time sequence data in the memory queue;
The statistical analysis module is used for carrying out statistical calculation on the time sequence data in the memory queue according to five minutes by utilizing the statistical analysis function of the statistical analysis callback function to obtain a statistical analysis result; when new time sequence data is acquired, performing history compensation operation;
The history back-filling operation includes: when new time sequence data is acquired, triggering a corresponding event, informing a related statistical analysis function to perform a compensation operation, checking whether the time sequence data is updated or not in a fixed time interval, and informing the related statistical analysis function to perform the compensation operation.
9. An electronic device comprising a memory for storing a computer program and a processor that runs the computer program to cause the electronic device to perform the method of extracting data characteristics of the time-series database of any one of claims 1 to 7.
10. The electronic device of claim 9, wherein the memory is a readable storage medium.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410058101.8A CN117891853A (en) | 2024-01-15 | 2024-01-15 | Data characteristic extraction method and system of time sequence database and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410058101.8A CN117891853A (en) | 2024-01-15 | 2024-01-15 | Data characteristic extraction method and system of time sequence database and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117891853A true CN117891853A (en) | 2024-04-16 |
Family
ID=90644124
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410058101.8A Pending CN117891853A (en) | 2024-01-15 | 2024-01-15 | Data characteristic extraction method and system of time sequence database and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117891853A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180293284A1 (en) * | 2017-04-10 | 2018-10-11 | Servicenow, Inc. | Systems and methods for querying time series data |
| WO2019009773A1 (en) * | 2017-07-07 | 2019-01-10 | Telefonaktiebolaget Lm Ericsson (Publ) | Methods, systems, databases and network nodes of data communication networks for handling data posts |
| CN110046183A (en) * | 2019-04-16 | 2019-07-23 | 北京易沃特科技有限公司 | A kind of time series data polymerization search method, equipment and medium |
| CN113297278A (en) * | 2021-07-26 | 2021-08-24 | 阿里云计算有限公司 | Time sequence database, data processing method, storage device and computer program product |
| CN113761013A (en) * | 2020-06-05 | 2021-12-07 | 杭州海康威视数字技术股份有限公司 | Time sequence data pre-statistical method and device and storage medium |
| CN116501734A (en) * | 2023-04-07 | 2023-07-28 | 武汉一格空间科技有限公司 | Real-time post-processing method and device for sensor time sequence data and electronic equipment |
-
2024
- 2024-01-15 CN CN202410058101.8A patent/CN117891853A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180293284A1 (en) * | 2017-04-10 | 2018-10-11 | Servicenow, Inc. | Systems and methods for querying time series data |
| WO2019009773A1 (en) * | 2017-07-07 | 2019-01-10 | Telefonaktiebolaget Lm Ericsson (Publ) | Methods, systems, databases and network nodes of data communication networks for handling data posts |
| CN110046183A (en) * | 2019-04-16 | 2019-07-23 | 北京易沃特科技有限公司 | A kind of time series data polymerization search method, equipment and medium |
| CN113761013A (en) * | 2020-06-05 | 2021-12-07 | 杭州海康威视数字技术股份有限公司 | Time sequence data pre-statistical method and device and storage medium |
| CN113297278A (en) * | 2021-07-26 | 2021-08-24 | 阿里云计算有限公司 | Time sequence database, data processing method, storage device and computer program product |
| CN116501734A (en) * | 2023-04-07 | 2023-07-28 | 武汉一格空间科技有限公司 | Real-time post-processing method and device for sensor time sequence data and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2021212756A1 (en) | Index anomaly analysis method and apparatus, and electronic device and storage medium | |
| US10248528B2 (en) | System monitoring method and apparatus | |
| CN105071983B (en) | Abnormal load detection method for cloud calculation on-line business | |
| CN114285728B (en) | Predictive model training method, traffic prediction device and storage medium | |
| CN113381890A (en) | Alarm information association method and device, electronic equipment and readable storage medium | |
| CN112380044B (en) | Data anomaly detection method, device, computer equipment and storage medium | |
| CN112907064B (en) | Electric quantity prediction method and device based on adaptive window, storage medium and terminal | |
| US20220245010A1 (en) | Time-series anomaly detection using an inverted index | |
| CN112199252A (en) | Abnormity monitoring method and device and electronic equipment | |
| CN117421994A (en) | Edge application health monitoring method and system | |
| CN117891853A (en) | Data characteristic extraction method and system of time sequence database and electronic equipment | |
| CN117236571A (en) | Planning method and system based on Internet of things | |
| Wang et al. | Clustering enabled few-shot load forecasting | |
| CN113610503B (en) | Resume information processing method, device, equipment and medium | |
| CN115456168A (en) | Training method and energy consumption determination method and device for reinforcement learning model | |
| EP4027277A1 (en) | Method, system and computer program product for drift detection in a data stream | |
| CN111771195A (en) | Stream processing apparatus and data stream processing method | |
| CN116257554A (en) | Demand statistical analysis method, device, equipment and medium based on virtual summary table | |
| CN113329128A (en) | Traffic data prediction method and device, electronic equipment and storage medium | |
| Wu et al. | A distributed real-time data prediction framework for large-scale time-series data using stream processing | |
| Yang et al. | Root Cause Location Based on Prophet and Kernel Density Estimation | |
| Guo et al. | Research on Intelligent Network Operation Management System Based on Anomaly Detection and Time Series Forecasting Algorithms | |
| CN117112245B (en) | Multithreading synchronous creation method and system based on equipment linkage | |
| Lupenko et al. | Approach to gas consumption process forecasting on the basis of a mathematical model in the form of a random cyclic process | |
| CN118444239B (en) | Ammeter metering method of intelligent ammeter and intelligent ammeter |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |