CN110688362A - Data sectional type storage method based on time stamp - Google Patents

Abstract

The invention discloses a data sectional type storage method based on a time stamp, which comprises the following steps: determining a time interval A to be stored, wherein A is less than or equal to 1 day, A is a second-level unit, i is a serial number, and the initial value of i is 1; when the time stamp is Ti, recording real-time data Ti of Ti: and X, when the time stamp is Ti +1, recording real-time data Ti +1 of Ti + 1: x, Ti +1 ═ Ti + a, difference data Xi within the time interval a are calculated, Xi ═ Ti + 1: X-Ti: x, deletion of Ti: x, storing the Xi into an array X; and inquiring difference data of a certain time according to the time stamp or the time point. The invention has the characteristics of higher query efficiency than the traditional storage method, greatly saving storage space and reducing query threshold.

Description

Data sectional type storage method based on time stamp

Technical Field

The invention relates to the technical field of data storage of energy management products, in particular to a data sectional type storage method based on a timestamp.

Background

Time stamping: the time stamp refers to the total seconds from the greenwich time 1970, 01, 00 hours and 00 minutes 00 seconds (from the time 1970, 01, 08 hours and 00 seconds) to the present, and it should be noted that 8 hours, i.e. 28800 seconds should be added to the timestamp when calculating the time of beijing.

Redis: the system is an open-source, network-supporting, memory-based and persistent log-type and Key-Value database, and provides APIs of multiple languages.

Real-time energy consumption: the absolute value of the difference between two acquired data in one acquisition period of the electric meter after subtraction is used, for example, the reading of the electric meter is acquired every 10 minutes, and then the energy consumption of 10 minutes is the real-time energy consumption of the electric meter.

The traditional meter counting data storage mode generally comprises methods of sequential storage or index storage and the like, wherein the sequential storage method is to store and collect real-time values according to the collection time sequence, the data is generally original data, and the later-stage calculation efficiency is low; the index storage method is to store the difference Value of a certain break time in a Key-Value (Key Value pair) form, where Key may be a certain time period, and Value is a data difference Value of a certain time period, and this way, when a large number of points are collected, the storage capacity is large, and the query consumes time.

Disclosure of Invention

The invention aims to overcome the defects of large storage amount, time consumption of query, low calculation efficiency and fuzzy query when meter data are stored for a certain period of time in the prior art, and provides a data sectional type storage method based on a timestamp.

In order to achieve the purpose, the invention adopts the following technical scheme:

a data segmented storage method based on time stamps comprises the following steps:

determining a time interval A to be stored, wherein A is less than or equal to 1 day, A is a second-level unit, i is a serial number, and the initial value of i is 1;

(1-1) recording real-time data Ti: X of Ti when the timestamp is Ti, recording real-time data T (i + 1): X of T (i +1) and T (i + 1): Ti + A when the timestamp is T (i +1), calculating difference data Xi in a time interval A, wherein Xi is T (i + 1): X-Ti: X, deleting Ti: X, and storing Xi in an array X;

(1-2) when i is less than n, increasing i by 1 and returning to the step (1-1); when i is more than or equal to n, finishing the collection work for 1 day, and turning to the step (1-3);

and (1-3) inquiring difference data of a certain time according to the time stamp or the time point.

The invention includes determining the time interval of segment storage, (not more than one day), then segmenting the time stamp of one day according to the determined interval, recording the current accumulated value at the beginning time of each segment, calculating the difference value with the accumulated value recorded last time, storing the difference value into the real-time database or the historical database, then the data stored in the database will be a continuous array segmented by time in one day, if the difference data of a specific time is needed to be obtained, the index of the previous continuous array can be obtained only by dividing the time interval of the segment by the time stamp of the day.

Firstly, aiming at large storage capacity and time consumption of query, the data in one point and one day are stored in the same field in a segmented mode, so that the storage unit is saved, the database overhead is reduced, the query access times are avoided, and the query time is optimized. And aiming at low calculation efficiency, because the data stored by the invention is the calculated difference, the additional calculation on the original data is avoided, namely, the calculation efficiency is improved. In addition, the invention solves the problem of fuzzy query of query time points, and data stored in a certain time can be indexed as long as the time point is in a certain time range.

Preferably, the query by timestamp comprises the steps of:

performing modulus calculation on 86400 seconds of 1 day by using a timestamp T1 needing to be queried to obtain a remainder which is Tx1 seconds from 0 point to query time of the day; then Tx1 is used to divide the time interval A completely, if the remainder is discarded, the obtained quotient C1 is the index of the inquired data in the array X, and the C1 th data in the array X is taken out as the inquired data.

Preferably, the query at a time point comprises the following steps:

time points are firstly decomposed into Mth minute and Sth second of the H hour of the day according to minutes and seconds; then, calculating the seconds from 0 point to the query time point on the day T2 by using a formula T2 which is H multiplied by 60+ M multiplied by 60+ S, and performing modulo calculation on the seconds from 1 day 86400 by T2 to obtain the remainder which is the seconds from 0 point to the query time on the day Tx 2; then Tx2 is used to divide the time interval A completely, if the remainder is discarded, the obtained quotient C2 is the index of the inquired data in the array X, and the C2 th data in the array X is taken out as the inquired data.

Therefore, the invention has the following beneficial effects: when the real-time energy consumption, the minute level energy consumption or the hour level energy consumption is calculated, the query efficiency is higher than that of the traditional storage method because the energy consumption value is stored instead of the original data; the segmented storage is adopted, when the data is stored in a Key-Value database, all the data are stored in one Value, so that the storage space is greatly saved, and the method is superior to the traditional storage method; the query time does not need to be accurate acquisition time, and the result can be fuzzily found only by being close to the acquisition time, so that the query threshold is reduced compared with the traditional mode.

Drawings

FIG. 1 is a schematic diagram of the time stamping, time interval, real-time data and segmented data of the present invention;

FIG. 2 is a schematic diagram of a storage manner of the segment data according to the present invention;

FIG. 3 is a flow chart of storing data according to the present invention;

FIG. 4 is a flow chart of querying data by time stamp in accordance with the present invention;

FIG. 5 is a flow chart of querying data at a point in time according to the present invention.

Detailed Description

The invention is further described with reference to the following figures and detailed description.

As shown in fig. 1, a time interval a to be stored is determined (i.e. when the number of meter counts in the time period a needs to be stored, a is not more than one day, i.e. 86400 seconds), a is a unit of second class, when the time stamp is T1, currently acquired real-time data T1: X is recorded, when the time reaches T2, i.e. the a-segment time is passed, difference data in the a-segment time, i.e. X1: T2: X-T1: X, is obtained, real-time data T2: X of T2 time is recorded, T1: X does not need to be reserved, and X1 is finally inserted into the array X stored in segments.

As shown in fig. 2, the steps in fig. 1 are continuously executed to obtain an array X of segmented data, which includes data X1 to Xn, and then the array X is stored in the historical database or the real-time database, where the flow of the whole storage process is shown in fig. 3.

When differential data for a certain period of time is to be queried, there are two methods: querying by timestamp and querying by point in time. Inquiring according to the time stamp, as shown in fig. 4, modulo 86400 seconds of a day by using a time stamp T1 needing to be inquired, obtaining a remainder as Tx1 from 0 point of the day to the inquiry time, dividing the time interval a by Tx1, if the remainder is discarded, obtaining a quotient C1 which is an index of the inquired data in the segmented array, and taking out C1 th data in the array, namely the inquired data, namely formula 1: index n1 ═ (T% 86400) | a, note: '%' is a modulus symbol in mathematics, and'|' is an integer division symbol in mathematics.

Inquiring according to the time point, as shown in fig. 5, firstly, decomposing the time point according to the time (H) and the minute (M) for the second (S), then obtaining the seconds T2 from the point 0 to the inquiring time point in the current day by using the formula H × 60 × 60+ M × 60+ S, wherein the following calculation is similar to the inquiring mode according to the timestamp, T2 divides the time interval a completely, if the remainder is discarded, the obtained quotient C2 is the index of the inquired data in the segmented array, and the C2 th data in the array is taken out, namely the inquired data, namely the formula 2: index n2 ═ (H × 60 × 60+ M × 60+ S) | a, note: '×' is a mathematical multiplication symbol, '+' is a mathematical addition symbol, and'|' is a mathematical division symbol.

For example:

and (3) storing: calculating the real-time energy consumption of a certain subway station, acquiring the real-time recording electricity degree reading of the total household electricity meter of the subway station, and assuming that the real-time energy consumption is the difference of the electricity degrees acquired twice every 10 minutes if the real-time energy consumption is acquired once every 10 minutes. Then the storage time is first determined to be 10 minutes, i.e. 600(10 x 60) seconds. Assuming that the first acquisition time is 0 min 0 s at 0 o' clock of 08/10/2019, the conversion time is 1565395200 (note: a conversion method is provided in programming language, the conversion is not needed, the beijing time is the east-eight area, 28800 should be added when the timestamp is calculated, i.e. 8 hours), and assuming that the acquired electricity reading is 1000. At this time, the reading of the electricity is recorded, the second collecting time is 0 point 10 minutes and 0 seconds at 10 days of 08 months and 10 months in 2019, the timestamp is 1565395800, and the collected electricity reading is 1700. At this time, the newly acquired data 1700 is used for subtracting the reading 1000 of the last time to obtain the energy consumption of 700kW · h, 700 is used as a first Value and stored as an array M, the acquired electric meter name is used as Key, the array a is used as Value and stored in Redis (note that the array a is used as Value and stored, can be converted into a character string, and is used as a separator and stored directly as an array), and finally the current acquisition Value 1700 is recorded, and the time of the third acquisition is the same as the time of the second acquisition, but only the energy consumption Value obtained by subtraction is used as a second element of the array and stored in Redis needed. And repeating the steps until all data of one day are acquired and stored, and taking the acquisition once every 10 minutes as an example, 144 energy consumption values should exist in the array.

And (3) inquiring: for example, the energy consumption value corresponding to 12 o' clock 12 min 50 sec at 08, 10 and 2019 needs to be inquired. According to the query by the timestamp, firstly, the timestamp corresponding to the query time is 1565439170, the query formula 1 is applied, T is 1565439170, a is 600, then the index n1 is 1565439170% 86400|600 is 73, and finally, the 73 th value in the array is the real-time energy consumption value of the query time.

And (3) inquiring the energy consumption value of 12 points, 12 points and 50 seconds at 20 points, 08 months, 10 months and 10 days in 9 years by using the formula 2, wherein H is 12, M is 12, S is 50 and A is 600, then the index n2 is (12 multiplied by 60+12 multiplied by 60+50) |600 is 73, and finally the 73 th value is the energy consumption value of the inquiry time, which is the same as the value obtained by the first method.

Time interval: the difference in seconds from one acquisition time point T1 to the next acquisition time point T2 during the day is the currently determined time interval.

Real-time data: the data acquired at one acquisition time point T is called real-time data.

Segmentation data: the difference between the data acquired at the end time point and the data acquired at the start time point of each time interval is referred to as segmentation data.

Segmented storage: each segmented data is inserted into an array in a sequential manner, and the array is stored in a database.

Indexing: the sort order number of each piece of segmented data in the stored array is the index of the data in the array.

And (3) time stamp query: and inquiring the real-time data difference value of the two acquisition time points of the time stamp by using the time stamp, namely the total seconds from 1970, 01, 00 hours and 00 minutes 00 seconds (Beijing time, 1970, 01, 08, 00 minutes and 00 seconds) to the inquiry time.

And (3) inquiring time points: and inquiring the data difference value of the two acquisition time points before and after the time point by using the time point such as 18:30: 22.

And (4) performing modulo calculation on the time interval by using the timestamp, wherein the timestamp is the next acquisition point when the judgment result is 0, and otherwise, the timestamp is not the next acquisition time point.

When a historical database or a real-time database is stored, one field stores the name of the acquired data point, and the other field stores an array of segmented data, so that one record in the database is an array of data differences calculated according to fixed time intervals in one day of the acquired data point name.

When data is queried, a segmented storage array stored on a day corresponding to a certain collected data point name needs to be queried, then an index of the queried data in the array can be obtained through a timestamp or time point querying method, and further the data in the array is taken out through the index.

It should be understood that this example is for illustrative purposes only and is not intended to limit the scope of the present invention. Further, it should be understood that various changes or modifications of the present invention may be made by those skilled in the art after reading the teaching of the present invention, and such equivalents may fall within the scope of the present invention as defined in the appended claims.

Claims (3)

1. A data segmented storage method based on time stamps is characterized by comprising the following steps:

determining a time interval A to be stored, wherein A is less than or equal to 1 day, A is a second-level unit, i is a serial number, and the initial value of i is 1;

(1-1) recording real-time data Ti of Ti when the time stamp is Ti: x, when the time stamp is T (i +1), recording the real-time data T (i +1) of T (i + 1): x, T (i +1) ═ Ti + a, difference data Xi in the time interval a are calculated, Xi ═ T (i + 1): X-Ti: x, deletion of Ti: x, storing the Xi into an array X;

(1-2) when i is less than n, increasing i by 1 and returning to the step (1-1); when i is more than or equal to n, finishing the collection work for 1 day, and turning to the step (1-3);

and (1-3) inquiring difference data of a certain time according to the time stamp or the time point.

2. The method of claim 1, wherein the querying by time stamp comprises the steps of:

performing modulus calculation on 86400 seconds of 1 day by using a timestamp T1 needing to be queried to obtain a remainder which is Tx1 seconds from 0 point to query time of the day; then Tx1 is used to divide the time interval A completely, if the remainder is discarded, the obtained quotient C1 is the index of the inquired data in the array X, and the C1 th data in the array X is taken out as the inquired data.

3. The method for storing data segment based on time stamp as claimed in claim 1 or 2, wherein the query by time point comprises the following steps:

time points are firstly decomposed into Mth minute and Sth second of the H hour of the day according to minutes and seconds; then, calculating the seconds from 0 point to the query time point on the day T2 by using a formula T2 which is H multiplied by 60+ M multiplied by 60+ S, and performing modulo calculation on the seconds from 1 day 86400 by T2 to obtain the remainder which is the seconds from 0 point to the query time on the day Tx 2; then Tx2 is used to divide the time interval A completely, if the remainder is discarded, the obtained quotient C2 is the index of the inquired data in the array X, and the C2 th data in the array X is taken out as the inquired data.

Images