CN117278046B - Time sequence data compression storage method and device, electronic equipment and storage medium - Google Patents
Time sequence data compression storage method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN117278046B CN117278046B CN202311211940.0A CN202311211940A CN117278046B CN 117278046 B CN117278046 B CN 117278046B CN 202311211940 A CN202311211940 A CN 202311211940A CN 117278046 B CN117278046 B CN 117278046B
- Authority
- CN
- China
- Prior art keywords
- data
- compression
- coding
- time sequence
- determining
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 32
- 238000013144 data compression Methods 0.000 title claims abstract description 23
- 238000007906 compression Methods 0.000 claims abstract description 265
- 230000006835 compression Effects 0.000 claims abstract description 265
- 238000011156 evaluation Methods 0.000 claims abstract description 69
- 239000012634 fragment Substances 0.000 claims description 12
- 238000010606 normalization Methods 0.000 claims description 9
- 238000004590 computer program Methods 0.000 claims description 7
- 238000012545 processing Methods 0.000 claims description 6
- 238000006467 substitution reaction Methods 0.000 claims description 5
- 238000012544 monitoring process Methods 0.000 claims description 4
- 238000013500 data storage Methods 0.000 abstract description 9
- 238000010586 diagram Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000000265 homogenisation Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000001356 surgical procedure Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H03—ELECTRONIC CIRCUITRY
- H03M—CODING; DECODING; CODE CONVERSION IN GENERAL
- H03M7/00—Conversion of a code where information is represented by a given sequence or number of digits to a code where the same, similar or subset of information is represented by a different sequence or number of digits
- H03M7/30—Compression; Expansion; Suppression of unnecessary data, e.g. redundancy reduction
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Compression, Expansion, Code Conversion, And Decoders (AREA)
Abstract
The invention discloses a time sequence data compression storage method, a device, electronic equipment and a storage medium, wherein the method comprises the following steps: acquiring time sequence data to be stored and a time stamp field, acquiring reference time sequence data matched with the time stamp field from a database, determining a data increment of the time sequence data to be stored, determining a data type of the data increment according to a historical access amount, encoding and compressing the data increment through a first encoding mode and a first compression mode if the data type is cold data to generate first target compressed data, determining a coding compression scheme with the highest evaluation score from a plurality of coding compression schemes supported by the database as a target coding compression scheme if the data type is hot data, and encoding and compressing the data increment based on the target coding compression scheme to generate second target compressed data, wherein each coding compression scheme comprises the encoding mode and the compression mode. Thereby saving storage space for time sequence data storage in the internet of things equipment.
Description
Technical Field
The present invention relates to the field of data storage, and in particular, to a time-series data compression storage method, apparatus, electronic device, and storage medium.
Background
In the related art, time sequence data collected by the internet of things equipment refer to data collected according to time dimension in a corresponding time stamp, the data comprise daily operation data, maintenance data and the like of the detected main equipment, and the data describe measured values of the main equipment at each moment in a time range. The time sequence data is generally stored by arranging a storage database in the Internet of things equipment, and the storage space needs to achieve the characteristics of large storage capacity, high reading and writing speed and the like so as to support the characteristics of continuous high concurrency, massive data writing, general no updating and low-delay query of the time sequence data. For the internet of things (THE INTERNET of Things, ioT) edge device, due to limited device resources, time-series data in the database is provided with timeliness, and the traditional data compression storage mode cannot meet the storage requirement of the time-series data in the internet of things edge device, so that the time-series data is easy to lose.
Disclosure of Invention
Aiming at the technical problem of high time sequence data storage cost of Internet of things equipment in the prior art, the invention provides a time sequence data compression storage method, a time sequence data compression storage device, electronic equipment and a storage medium.
In order to achieve the above purpose, the invention is realized by the following technical scheme:
in a first aspect of the embodiment of the present invention, there is provided a surgical navigation method for percutaneous puncture surgery, applied to an internet of things device, the method comprising:
acquiring time sequence data to be stored and a time stamp field acquired by the Internet of things equipment in a preset time range;
acquiring reference time sequence data matched with the time stamp field from a database, comparing the time sequence data to be stored with the reference time sequence data, and determining the data increment of the time sequence data to be stored;
determining a historical access amount of historical storage data under a historical timestamp field corresponding to the timestamp field in the database, and determining a data type of the data increment according to the historical access amount, wherein the data type comprises cold data and hot data;
If the data type is the cold data, encoding and compressing the data increment through a first encoding mode and a first compression mode to generate first target compressed data;
And if the data type is the hot data, determining a coding compression scheme with the highest evaluation score from a plurality of coding compression schemes supported by the database as a target coding compression scheme, and coding and compressing the data increment based on the target coding compression scheme to generate second target compression data, wherein each coding compression scheme comprises a coding mode and a compression mode.
Optionally, the determining, from the multiple coding compression schemes supported by the database, the coding compression scheme with the highest evaluation score as the target coding compression scheme includes:
Randomly combining the plurality of coding compression schemes to generate a plurality of first coding compression schemes;
Determining a plurality of compression evaluation parameters corresponding to the plurality of first coding compression schemes one by one, wherein each compression evaluation parameter comprises at least one of a compression rate parameter, a compression time parameter, a coding time parameter and a decoding time parameter;
Determining a first evaluation score corresponding to a first encoding compression scheme based on any one of the plurality of compression evaluation parameters, traversing the plurality of first encoding compression schemes, and generating a plurality of evaluation scores;
And determining a coding compression scheme with the highest evaluation score from the plurality of first coding compression schemes as the target coding compression scheme based on the plurality of evaluation scores.
Optionally, the determining, based on any first compression evaluation parameter of the plurality of compression evaluation parameters, a first evaluation score corresponding to a first coding compression scheme includes:
Acquiring a plurality of weight values of a plurality of compression parameters in the first compression evaluation parameters;
normalizing the plurality of compression parameters to generate a plurality of target compression parameters;
The first evaluation score of the first encoding compression scheme is determined according to the plurality of weight values and the plurality of target compression parameters.
Optionally, the normalizing the plurality of compression parameters generates a plurality of target compression parameters, including:
For any one compression parameter of the plurality of compression parameters, normalizing the compression parameter by the following formula:
The compression parameter normalization method comprises the steps of determining a compression parameter, wherein X normalized is a target compression parameter after normalization of the compression parameter, X is the compression parameter, X mean is the average value of the compression parameters, and X std is the standard deviation value of the compression parameters.
Optionally, the encoding and compressing the data increment by the first encoding mode and the first compression mode to generate first target compressed data includes:
acquiring frequency data of each character in the data increment based on the first coding mode;
Constructing a weight binary tree according to the frequency data, wherein characters with low frequency in the frequency data are positioned at the lower layer of the weight binary tree, and characters with high frequency in the frequency data are positioned at the upper layer of the weight binary tree;
determining the coding information of each character according to the position of each character in the weight binary tree;
Performing coding substitution on the data increment according to the coding information to generate target coding information;
And compressing the target coding information in the first compression mode to generate the first target compressed data.
Optionally, the determining the coding information of each character according to the position of each character in the weight binary tree includes:
Determining a plurality of paths from a root node to the respective characters in the weight binary tree;
and encoding the paths based on a preset rule to generate the encoding information of each character.
Optionally, the acquiring the time sequence data to be stored and the timestamp field acquired by the internet of things device in the preset time range includes:
monitoring the running state of the Internet of things equipment to obtain a plurality of initial time sequence data to be stored;
Performing slicing processing on the plurality of initial time sequence data points to be stored to obtain a plurality of time sequence data fragments;
determining the traversing sequence of different time sequence data in each time sequence data segment according to the time stamp field;
invoking multithreading to synchronously traverse the time sequence data of the initial time sequence data fragments;
and acquiring the time sequence data to be stored in the current traversal sequence in the plurality of initial time sequence data fragments.
According to a second aspect of embodiments of the present disclosure, there is provided a time-series data compression storage apparatus applied to an internet of things device, the apparatus including:
the acquisition module is used for acquiring time sequence data to be stored and a time stamp field acquired by the Internet of things equipment in a preset time range;
The first determining module is used for acquiring reference time sequence data matched with the time stamp field from a database, comparing the time sequence data to be stored with the reference time sequence data and determining the data increment of the time sequence data to be stored;
A second determining module, configured to determine a historical access amount of historical storage data under a historical timestamp field corresponding to the timestamp field in the database, and determine a data type of the data increment according to the historical access amount, where the data type includes cold data and hot data;
The first execution module is used for encoding and compressing the data increment through a first encoding mode and a first compression mode if the data type is the cold data so as to generate first target compressed data;
And the second execution module is used for determining the coding compression scheme with the highest evaluation score from a plurality of coding compression schemes supported by the database as a target coding compression scheme if the data type is the hot data, and coding and compressing the data increment based on the target coding compression scheme to generate second target compressed data, wherein each coding compression scheme comprises a coding mode and a compression mode.
According to a third aspect of embodiments of the present disclosure, there is provided an electronic device, comprising:
A processor;
A memory for storing processor-executable instructions;
wherein the processor is configured to execute the executable instructions to implement the steps of the time series data compression storage method of any one of the first aspects of the embodiments of the present disclosure.
According to a fourth aspect of embodiments of the present disclosure, there is provided a computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the steps of the time series data compression storage method of any one of the first aspects of embodiments of the present disclosure.
The invention provides a time sequence data compression storage method, a time sequence data compression storage device, electronic equipment and a storage medium, which have the following beneficial effects compared with the prior art:
According to the method, the time sequence data to be stored and the time stamp field which are acquired by the Internet of things equipment in the preset time range are acquired, the reference time sequence data matched with the time stamp field is acquired from the database, the time sequence data to be stored is compared with the reference time sequence data, the data increment of the time sequence data to be stored is determined, the historical access amount of the historical storage data under the historical time stamp field corresponding to the time stamp field in the database is determined, the data type of the data increment is determined according to the historical access amount, the data type comprises cold data and hot data, if the data type is the cold data, the data increment is encoded and compressed through a first encoding mode and a first compression mode to generate first target compression data, if the data type is the hot data, the encoding compression mode with the highest evaluation score is determined from a plurality of encoding compression modes supported by the database to serve as target encoding compression modes, the data increment is encoded and compressed based on the target encoding compression modes to generate second target compression data, and each encoding compression mode comprises the encoding mode and the compression mode. Therefore, the storage pressure of time sequence data in the Internet of things equipment is reduced in a data increment mode, different coding compression schemes are distributed according to the data type of the time sequence data, the storage space of the time sequence data is further compressed, and the storage space is saved for time sequence data storage in the Internet of things equipment.
Drawings
Fig. 1 is a flow chart illustrating a method of time-series data compression storage according to an exemplary embodiment.
Fig. 2 is a flow chart illustrating a method of determining a target encoding compression scheme according to an exemplary embodiment.
Fig. 3 is a flowchart illustrating a method of generating first target compressed data according to an exemplary embodiment.
FIG. 4 is a block diagram illustrating a time series data compression storage device according to an example embodiment.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
An embodiment of the present invention provides a time-series data compression storage method, fig. 1 is a flowchart of a time-series data compression storage method according to an exemplary embodiment, and as shown in fig. 1, the method is applied to an internet of things device, and the method includes the following steps.
Step S11, acquiring time sequence data to be stored and a time stamp field acquired by the Internet of things equipment within a preset time range.
The internet of things equipment is usually edge equipment and is arranged at the far end of the principle server, the running data are monitored through the running state of the internet of things equipment, and the running state and the running data are transmitted to the server so that the server can analyze the current state of the internet of things equipment based on the data. It can be understood that the internet of things device does not upload the time sequence data in real time, the time sequence data generated in the device is stored first, and after the server issues the inquiry uploading instruction, the time sequence data stored in the internet of things device is transmitted to the server. The storage space of the internet of things equipment is limited, and the data storage of the internet of things equipment cannot be met by a common data storage mode based on the high-frequency property and the richness of the time sequence data, so that the proposal provides a compression storage method of the time sequence data to save the storage space of the time sequence data in the internet of things equipment.
The method and the device are applied to Internet of things equipment, the Internet of things equipment collects time sequence data in a preset time range, and a time stamp field for generating the time sequence data to be stored is recorded. In this embodiment, the internet of things device records operation data of a corresponding device, determines time sequence data to be stored in a preset time range, where the time sequence data to be stored may be operation record data of the internet of things device, and the like.
Alternatively, in one embodiment, step S11 includes:
monitoring the running state of the Internet of things equipment to obtain a plurality of initial time sequence data to be stored;
Performing slicing processing on the plurality of initial time sequence data points to be stored to obtain a plurality of time sequence data fragments;
determining the traversing sequence of different time sequence data in each time sequence data segment according to the time stamp field;
invoking multithreading to synchronously traverse the time sequence data of the initial time sequence data fragments;
and acquiring the time sequence data to be stored in the current traversal sequence in the plurality of initial time sequence data fragments.
Step S12, obtaining the reference time sequence data matched with the time stamp field from a database, comparing the time sequence data to be stored with the reference time sequence data, and determining the data increment of the time sequence data to be stored.
In an example, in the embodiment, a plurality of types of reference time series data under a plurality of different time stamp fields are stored in the internet of things device in advance, and in an initial state, based on the time series data initially stored in the internet of things device, a time stamp field of the time series data is determined, and the time series data is used as the reference time series data under the time stamp field. And inquiring the reference time sequence data matched with the time stamp field from the internet equipment through the time stamp field of the current time sequence data to be stored.
Comparing the reference time sequence data with the time sequence data to be stored, and determining the difference of the time sequence data to be stored relative to the reference time sequence data, wherein the difference comprises the data increment and the data variation of the time sequence data to be stored, and determining the data increment of the time sequence data to be stored through comparison. It can be understood that, in this embodiment, the time series data to be stored is mainly operation data of the internet of things device, and on the premise of no special exception, the contemporaneous time series data of the internet of things device basically does not change too much, if the contemporaneous time series data of the internet of things device are collected and stored in the same manner, the corresponding storage obtains higher homogenization degree of the time series data, and the meaning of the operation condition of the internet of things device is lower, so that the change amount of the time series data can be stored, the time series data of the period can be deduced based on the change amount between the contemporaneous data and the reference data, and meanwhile, the storage space of the time series data in the internet of things device can be reduced.
Step S13, determining a historical access amount of historical storage data under a historical timestamp field corresponding to the timestamp field in the database, and determining a data type of the data increment according to the historical access amount, wherein the data type comprises cold data and hot data.
It can be understood that the access degree of the time series data under the same time stamp field in the internet of things device is the same, for example, the access amount of the time series data in the afternoon time period on the internet of things device is lower than the historical access amount of the time series data collected in the afternoon time period. Accordingly, the historical access amount of the historical stored data under the corresponding historical timestamp field can be referred to according to the timestamp field corresponding to the current time series data to be stored, so that the data type of the current time series data to be stored can be determined, wherein the data type comprises cold data and hot data. For example, if the historical access amount is greater than or equal to the set access amount threshold, determining that the data type of the time-series data to be stored is hot data; and if the historical access quantity is smaller than the set access quantity threshold value, determining the data type of the time sequence data to be stored as cold data. For example, if the acquisition time stamp field of the current time sequence data to be stored is 8 months 15 days 13:00, the historical access amount of the historical storage data acquired by the corresponding historical time stamp field is determined based on the time stamp field, the historical access amount can be the historical access times, the data type of the current time sequence data to be stored is determined according to the historical access times, if the historical access times are greater than 5 times, the current time sequence data to be stored is determined to be hot data, and if the historical access times are less than 5 times, the current time sequence data to be stored is determined to be cold data.
And step S14, if the data type is the cold data, encoding and compressing the data increment through a first encoding mode and a first compression mode to generate first target compressed data.
In this embodiment, the internet of things device is loaded with multiple coding compression schemes, where each coding compression scheme includes at least one coding mode and one compression mode. When the data type of the time sequence data to be stored is determined to be cold data, the data increment generated in the step is encoded and compressed through a first encoding mode and a first compression mode which are set in advance, and first target compressed data is generated. For example, in this embodiment, a bit compression manner may be adopted to perform encoding compression on a data increment of time-series data to be stored, and according to a preset rule corresponding to a first encoding manner, the data increment is first converted into an encoded character, and then the encoded character is stored based on the first compression manner, so as to generate first target compressed data.
And S15, if the data type is hot data, determining the coding compression scheme with the highest evaluation score from a plurality of coding compression schemes supported by a database as a target coding compression scheme, and coding and compressing the data increment based on the target coding compression scheme to generate second target compressed data, wherein each coding compression scheme comprises a coding mode and a compression mode.
For example, in the present embodiment, when it is determined that the data type of the time-series data to be stored is hot data, the encoding compression scheme having the highest evaluation score is determined as the target encoding compression scheme from among the plurality of encoding compression schemes supported by the database. In this embodiment, based on the specificity of the thermal data, the thermal data is compressed by adopting the coding compression scheme with the highest evaluation score, so that the thermal data is stored in an optimal coding compression mode while the thermal data is not lost, and the data storage space of the thermal data is further saved. The internet of things device stores various coding compression schemes, acquires factors such as coding compression time, compression rate, processor occupation proportion and the like of the coding compression schemes, comprehensively evaluates various coding compression schemes according to the parameters, and can evaluate the various coding compression schemes according to scores based on the parameters, wherein the higher the score is, the higher the corresponding coding compression scheme is evaluated.
For example, a target encoding compression scheme with the highest evaluation score is determined from a plurality of encoding compression schemes based on the steps, and then the data increment under the timestamp field is encoded and compressed based on the target encoding compression scheme to generate second target compressed data.
According to the method, the time sequence data to be stored and the time stamp field which are acquired by the Internet of things equipment in the preset time range are acquired, the reference time sequence data matched with the time stamp field is acquired from the database, the time sequence data to be stored is compared with the reference time sequence data, the data increment of the time sequence data to be stored is determined, the historical access amount of the historical storage data under the historical time stamp field corresponding to the time stamp field in the database is determined, the data type of the data increment is determined according to the historical access amount, the data type comprises cold data and hot data, if the data type is the cold data, the data increment is encoded and compressed through a first encoding mode and a first compression mode to generate first target compression data, if the data type is the hot data, the encoding compression mode with the highest evaluation score is determined from a plurality of encoding compression modes supported by the database to serve as target encoding compression modes, the data increment is encoded and compressed based on the target encoding compression modes to generate second target compression data, and each encoding compression mode comprises the encoding mode and the compression mode. Therefore, the storage pressure of time sequence data in the Internet of things equipment is reduced in a data increment mode, different coding compression schemes are distributed according to the data type of the time sequence data, the storage space of the time sequence data is further compressed, and the storage space is saved for time sequence data storage in the Internet of things equipment.
Fig. 2 is a flowchart illustrating a method for determining a target coding compression scheme according to an exemplary embodiment, and as shown in fig. 2, the step S15 includes:
step S151, randomly combining the multiple coding compression schemes to generate multiple first coding compression schemes.
Step S152, determining a plurality of compression evaluation parameters corresponding to the plurality of first coding compression schemes one by one, where each compression evaluation parameter includes at least one of a compression rate parameter, a compression time parameter, a coding time parameter, and a decoding time parameter.
Step S153, determining a first evaluation score corresponding to the first coding compression scheme based on any one of the compression evaluation parameters, traversing the first coding compression schemes, and generating a plurality of evaluation scores.
And step S154, determining the code compression scheme with the highest evaluation score from the first code compression schemes as the target code compression scheme based on the evaluation scores.
In this embodiment, a plurality of first encoding compression schemes are generated by randomly combining a plurality of encoding compression schemes stored in the internet of things device. A plurality of compression evaluation parameters of a plurality of first coding compression schemes are determined, wherein each coding compression scheme corresponds to one compression evaluation parameter, and each compression evaluation parameter comprises at least one of a compression rate parameter, a compression time parameter, a coding time parameter and a decoding time parameter. Determining a first evaluation score corresponding to a first coding compression scheme according to any first compression evaluation parameter in a plurality of compression evaluation parameters, traversing the plurality of first coding compression schemes to generate a plurality of evaluation scores, and determining a coding compression scheme with the highest evaluation score from the plurality of first coding compression schemes as the target coding compression scheme based on the plurality of evaluation scores.
Optionally, in one embodiment, step S153 includes:
Acquiring a plurality of weight values of a plurality of compression parameters in the first compression evaluation parameters;
normalizing the plurality of compression parameters to generate a plurality of target compression parameters;
The first evaluation score of the first encoding compression scheme is determined according to the plurality of weight values and the plurality of target compression parameters.
It can be understood that the evaluation values corresponding to different compression parameters are different, and some values are higher but the actual reflected evaluation degree is lower, so that normalization processing needs to be performed on each compression parameter to generate a plurality of target compression parameters, and then the first evaluation score of the coding compression scheme is determined according to the rated weight value of the target compression parameter.
Optionally, in an embodiment, the step of normalizing the plurality of compression parameters to generate a plurality of target compression parameters includes:
For any one compression parameter of the plurality of compression parameters, normalizing the compression parameter by the following formula:
The compression parameter normalization method comprises the steps of determining a compression parameter, wherein X normalized is a target compression parameter after normalization of the compression parameter, X is the compression parameter, X mean is the average value of the compression parameters, and X std is the standard deviation value of the compression parameters.
Fig. 3 is a flowchart illustrating a method for generating first target compressed data according to an exemplary embodiment, and as shown in fig. 3, the above step S14 includes the following steps.
Step S141, based on the first coding mode, obtaining frequency data of occurrence of each character in the data increment.
Step S142, a weight binary tree is constructed according to the frequency data, wherein characters with low frequency in the frequency data are located at the lower layer of the weight binary tree, and characters with high frequency in the frequency data are located at the upper layer of the weight binary tree.
And step S143, determining the coding information of each character according to the position of each character in the weight binary tree.
And step S144, performing coding substitution on the data increment according to the coding information to generate target coding information.
Step S145, compressing the target coding information by the first compression method, to generate the first target compressed data.
Illustratively, in this embodiment, the occurrence frequency of each symbol (such as a character or a byte) in the incremental data is counted, and a binary Tree of weights is constructed according to the frequency, and in this embodiment, the binary Tree of weights may be a hall Tree (Huffman Tree), where symbols with lower frequency are located at a lower layer of the Tree, and symbols with higher frequency are located at a higher layer of the Tree. The unique codes are assigned by 0 and 1 on the path from the root node to each symbol. A 0 on the path indicates left and a1 indicates right. The original data is replaced or appended according to the assigned encoding so that the data can be stored or transmitted in a more compact manner.
Optionally, in some embodiments, step S144 includes:
Determining a plurality of paths from a root node to the respective characters in the weight binary tree;
and encoding the paths based on a preset rule to generate the encoding information of each character.
In this embodiment, the multiple paths from the root node to each character in the weight binary tree are determined, and the value on the corresponding path is determined according to the trend of the path, if the paths are based on the lower paths of the same node, the node corresponding to the path is identified as 0 if the paths are to the left, and if the paths are to the right, the node corresponding to the path is identified as 1. In this way, a plurality of paths are labeled, and encoded information represented by each character is generated.
Fig. 4 is a block diagram of a time series data compression storage device according to an exemplary embodiment, as shown in fig. 4, the device 100 includes: the first determining module 120, the second determining module 130, the first executing module 140, and the second executing module 150.
An obtaining module 110, configured to obtain time sequence data to be stored and a timestamp field collected by the internet of things device within a preset time range;
A first determining module 120, configured to obtain, from a database, reference time sequence data that matches the timestamp field, compare the time sequence data to be stored with the reference time sequence data, and determine a data increment of the time sequence data to be stored;
A second determining module 130, configured to determine a historical access amount of historical stored data under a historical timestamp field corresponding to the timestamp field in the database, and determine a data type of the data increment according to the historical access amount, where the data type includes cold data and hot data;
The first execution module 140 is configured to encode and compress the data increment by a first encoding mode and a first compression mode to generate first target compressed data if the data type is the cold data;
And the second execution module 150 is configured to determine, if the data type is the hot data, a coding compression scheme with the highest evaluation score from multiple coding compression schemes supported by the database as a target coding compression scheme, and encode and compress the data increment based on the target coding compression scheme to generate second target compressed data, where each coding compression scheme includes an encoding mode and a compression mode.
Optionally, the second execution module 150 includes:
The generation submodule is used for randomly combining the plurality of coding compression schemes to generate a plurality of first coding compression schemes;
the first determining submodule is used for determining a plurality of compression evaluation parameters corresponding to the plurality of first coding compression schemes one by one, wherein each compression evaluation parameter comprises at least one of a compression rate parameter, a compression time parameter, a coding time parameter and a decoding time parameter;
a second determining sub-module, configured to determine a first evaluation score corresponding to a first encoding compression scheme based on any one of the plurality of compression evaluation parameters, traverse the plurality of first encoding compression schemes, and generate a plurality of evaluation scores;
And the execution submodule is used for determining the coding compression scheme with the highest evaluation score from the plurality of first coding compression schemes as the target coding compression scheme based on the plurality of evaluation scores.
Optionally, the second determining sub-module comprises:
An obtaining unit, configured to obtain a plurality of weight values of a plurality of compression parameters in the first compression evaluation parameter;
the generating unit is used for carrying out normalization processing on the plurality of compression parameters and generating a plurality of target compression parameters;
And the determining unit is used for determining the first evaluation score of the first coding compression scheme according to the weight values and the target compression parameters.
Optionally, the determining unit is configured to:
For any one compression parameter of the plurality of compression parameters, normalizing the compression parameter by the following formula:
The compression parameter normalization method comprises the steps of determining a compression parameter, wherein X normalized is a target compression parameter after normalization of the compression parameter, X is the compression parameter, X mean is the average value of the compression parameters, and X std is the standard deviation value of the compression parameters.
Optionally, the first execution module 140 includes:
The obtaining submodule is used for obtaining the frequency data of each character in the data increment based on the first coding mode;
a construction sub-module, configured to construct a weight binary tree according to the frequency data, where a character with a low frequency in the frequency data is located at a lower layer of the weight binary tree, and a character with a high frequency in the frequency data is located at a higher layer of the weight binary tree;
The determining submodule is used for determining the coding information of each character according to the position of each character in the weight binary tree;
The first generation sub-module is used for carrying out coding substitution on the data increment according to the coding information to generate target coding information;
And the second generation sub-module is used for compressing the target coding information through the first compression mode to generate the first target compressed data.
Optionally, the determining submodule is configured to:
Determining a plurality of paths from a root node to the respective characters in the weight binary tree;
and encoding the paths based on a preset rule to generate the encoding information of each character.
Optionally, the obtaining module 110 is configured to:
monitoring the running state of the Internet of things equipment to obtain a plurality of initial time sequence data to be stored;
Performing slicing processing on the plurality of initial time sequence data points to be stored to obtain a plurality of time sequence data fragments;
determining the traversing sequence of different time sequence data in each time sequence data segment according to the time stamp field;
invoking multithreading to synchronously traverse the time sequence data of the initial time sequence data fragments;
and acquiring the time sequence data to be stored in the current traversal sequence in the plurality of initial time sequence data fragments.
According to the method, the time sequence data to be stored and the time stamp field which are acquired by the Internet of things equipment in the preset time range are acquired, the reference time sequence data matched with the time stamp field is acquired from the database, the time sequence data to be stored is compared with the reference time sequence data, the data increment of the time sequence data to be stored is determined, the historical access amount of the historical storage data under the historical time stamp field corresponding to the time stamp field in the database is determined, the data type of the data increment is determined according to the historical access amount, the data type comprises cold data and hot data, if the data type is the cold data, the data increment is encoded and compressed through a first encoding mode and a first compression mode to generate first target compression data, if the data type is the hot data, the encoding compression mode with the highest evaluation score is determined from a plurality of encoding compression modes supported by the database to serve as target encoding compression modes, the data increment is encoded and compressed based on the target encoding compression modes to generate second target compression data, and each encoding compression mode comprises the encoding mode and the compression mode. Therefore, the storage pressure of time sequence data in the Internet of things equipment is reduced in a data increment mode, different coding compression schemes are distributed according to the data type of the time sequence data, the storage space of the time sequence data is further compressed, and the storage space is saved for time sequence data storage in the Internet of things equipment.
Embodiments of the present disclosure are also directed to an electronic device including:
a memory having a computer program stored thereon;
a processor for executing the computer program in the memory to implement the steps of the time-series data compression storage method according to any one of the above embodiments.
In another exemplary embodiment, a computer program product is also provided, comprising a computer program executable by a programmable apparatus, the computer program having code portions for performing the above described time-series data compression storage method when executed by the programmable apparatus.
With the above-described preferred embodiments according to the present application as a teaching, the worker skilled in the art could make various changes and modifications without departing from the scope of the technical idea of the present application. The technical scope of the present application is not limited to the contents of the specification.
Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made therein without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.
Claims (4)
1. The time sequence data compression storage method is characterized by being applied to the Internet of things equipment, and comprises the following steps:
acquiring time sequence data to be stored and a time stamp field acquired by the Internet of things equipment in a preset time range;
acquiring reference time sequence data matched with the time stamp field from a database, comparing the time sequence data to be stored with the reference time sequence data, and determining the data increment of the time sequence data to be stored;
determining a historical access amount of historical storage data under a historical timestamp field corresponding to the timestamp field in the database, and determining a data type of the data increment according to the historical access amount, wherein the data type comprises cold data and hot data;
If the data type is the cold data, encoding and compressing the data increment through a first encoding mode and a first compression mode to generate first target compressed data;
If the data type is the hot data, determining a coding compression scheme with the highest evaluation score from a plurality of coding compression schemes supported by the database as a target coding compression scheme, and coding and compressing the data increment based on the target coding compression scheme to generate second target compression data, wherein each coding compression scheme comprises a coding mode and a compression mode;
the determining, from among the plurality of coding compression schemes supported by the database, the coding compression scheme having the highest evaluation score as the target coding compression scheme includes:
Randomly combining the plurality of coding compression schemes to generate a plurality of first coding compression schemes;
Determining a plurality of compression evaluation parameters corresponding to the plurality of first coding compression schemes one by one, wherein each compression evaluation parameter comprises at least one of a compression rate parameter, a compression time parameter, a coding time parameter and a decoding time parameter;
determining a first evaluation score corresponding to a first encoding compression scheme based on any one of the plurality of compression evaluation parameters, traversing the plurality of first encoding compression schemes, and generating a plurality of evaluation scores;
Determining, based on the plurality of evaluation scores, a coding compression scheme having a highest evaluation score from the plurality of first coding compression schemes as the target coding compression scheme;
the determining a first evaluation score corresponding to a first encoding compression scheme based on any first compression evaluation parameter of the plurality of compression evaluation parameters includes:
Acquiring a plurality of weight values of a plurality of compression parameters in the first compression evaluation parameters;
normalizing the plurality of compression parameters to generate a plurality of target compression parameters;
Determining the first evaluation score of the first encoding compression scheme according to the plurality of weight values and the plurality of target compression parameters;
the normalizing the plurality of compression parameters to generate a plurality of target compression parameters includes:
For any one compression parameter of the plurality of compression parameters, normalizing the compression parameter by the following formula:
Wherein, X normalized is a target compression parameter after normalization of the compression parameter, X is the compression parameter, X mean is a mean value of the compression parameters, and X std is a standard deviation value of the compression parameters;
the encoding and compressing the data increment by a first encoding mode and a first compression mode to generate first target compressed data, including:
acquiring frequency data of each character in the data increment based on the first coding mode;
Constructing a weight binary tree according to the frequency data, wherein characters with low frequency in the frequency data are positioned at the lower layer of the weight binary tree, and characters with high frequency in the frequency data are positioned at the upper layer of the weight binary tree;
determining the coding information of each character according to the position of each character in the weight binary tree;
Performing coding substitution on the data increment according to the coding information to generate target coding information;
compressing the target coding information in the first compression mode to generate first target compressed data;
the determining the coding information of each character according to the position of each character in the weight binary tree comprises the following steps:
Determining a plurality of paths from a root node to the respective characters in the weight binary tree;
Encoding the paths based on a preset rule to generate the encoding information of each character;
the acquiring the time sequence data to be stored and the time stamp field acquired by the internet of things equipment within a preset time range comprises the following steps:
monitoring the running state of the Internet of things equipment to obtain a plurality of initial time sequence data to be stored;
performing slicing processing on the plurality of initial time sequence data to be stored to obtain a plurality of time sequence data fragments;
determining the traversing sequence of different time sequence data in each time sequence data segment according to the time stamp field;
invoking multithreading to synchronously traverse the time sequence data of the time sequence data fragments;
and acquiring the time sequence data to be stored in the current traversal sequence in the plurality of time sequence data fragments.
2. A time-series data compression storage device applied to an internet of things device and used for realizing the method as claimed in claim 1, wherein the device comprises:
the acquisition module is used for acquiring time sequence data to be stored and a time stamp field acquired by the Internet of things equipment in a preset time range;
The first determining module is used for acquiring reference time sequence data matched with the time stamp field from a database, comparing the time sequence data to be stored with the reference time sequence data and determining the data increment of the time sequence data to be stored;
A second determining module, configured to determine a historical access amount of historical storage data under a historical timestamp field corresponding to the timestamp field in the database, and determine a data type of the data increment according to the historical access amount, where the data type includes cold data and hot data;
The first execution module is used for encoding and compressing the data increment through a first encoding mode and a first compression mode if the data type is the cold data so as to generate first target compressed data;
And the second execution module is used for determining the coding compression scheme with the highest evaluation score from a plurality of coding compression schemes supported by the database as a target coding compression scheme if the data type is the hot data, and coding and compressing the data increment based on the target coding compression scheme to generate second target compressed data, wherein each coding compression scheme comprises a coding mode and a compression mode.
3. An electronic device, comprising:
A processor;
A memory for storing processor-executable instructions;
Wherein the processor is configured to execute the executable instructions to implement the steps of the time series data compression storage method of claim 1.
4. A computer readable storage medium having stored thereon computer program instructions, which when executed by a processor, implement the steps of the time series data compression storage method of claim 1.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311211940.0A CN117278046B (en) | 2023-09-18 | 2023-09-18 | Time sequence data compression storage method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311211940.0A CN117278046B (en) | 2023-09-18 | 2023-09-18 | Time sequence data compression storage method and device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117278046A CN117278046A (en) | 2023-12-22 |
| CN117278046B true CN117278046B (en) | 2024-06-11 |
Family
ID=89208737
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311211940.0A Active CN117278046B (en) | 2023-09-18 | 2023-09-18 | Time sequence data compression storage method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117278046B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117827957B (en) * | 2024-01-03 | 2024-06-11 | 广州安广电子科技股份有限公司 | Big data-based information input visualization method, system, medium and equipment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109981111A (en) * | 2019-03-28 | 2019-07-05 | 杭州即得科技有限公司 | A kind of method, apparatus and storage electronic equipment of data compression coding |
| CN110134723A (en) * | 2019-05-22 | 2019-08-16 | 网易（杭州）网络有限公司 | A kind of method and database of storing data |
| CN110633277A (en) * | 2019-08-13 | 2019-12-31 | 平安科技（深圳）有限公司 | Time sequence data storage method and device, computer equipment and storage medium |
| CN116680269A (en) * | 2023-06-09 | 2023-09-01 | 中冶赛迪信息技术(重庆)有限公司 | Time sequence data coding and compressing method, system, equipment and medium |
| CN116760661A (en) * | 2023-06-06 | 2023-09-15 | 厦门海辰储能科技股份有限公司 | Data storage method, apparatus, computer device, storage medium, and program product |
-
2023
- 2023-09-18 CN CN202311211940.0A patent/CN117278046B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109981111A (en) * | 2019-03-28 | 2019-07-05 | 杭州即得科技有限公司 | A kind of method, apparatus and storage electronic equipment of data compression coding |
| CN110134723A (en) * | 2019-05-22 | 2019-08-16 | 网易（杭州）网络有限公司 | A kind of method and database of storing data |
| CN110633277A (en) * | 2019-08-13 | 2019-12-31 | 平安科技（深圳）有限公司 | Time sequence data storage method and device, computer equipment and storage medium |
| CN116760661A (en) * | 2023-06-06 | 2023-09-15 | 厦门海辰储能科技股份有限公司 | Data storage method, apparatus, computer device, storage medium, and program product |
| CN116680269A (en) * | 2023-06-09 | 2023-09-01 | 中冶赛迪信息技术(重庆)有限公司 | Time sequence data coding and compressing method, system, equipment and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117278046A (en) | 2023-12-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11030157B2 (en) | Template based data reduction for commercial data mining | |
| CN117278046B (en) | Time sequence data compression storage method and device, electronic equipment and storage medium | |
| US11119834B2 (en) | Artificial creation of dominant sequences that are representative of logged events | |
| CN110633277B (en) | Time sequence data storage method, device, computer equipment and storage medium | |
| CN107368517A (en) | A kind of method and apparatus of high amount of traffic inquiry | |
| CN112286867B (en) | Oil-gas field time sequence data storage method, oil-gas field time sequence data query device and storage medium | |
| US7587388B2 (en) | Separating uploads into aggregate and raw data storage | |
| CN111709491A (en) | Anomaly detection method, device and equipment based on self-encoder and storage medium | |
| CN111190792B (en) | Log storage method and device, electronic equipment and readable storage medium | |
| CN110995273B (en) | Data compression method, device, equipment and medium for power database | |
| Xu et al. | An adaptive algorithm for online time series segmentation with error bound guarantee | |
| CN110569179A (en) | Data acquisition system and data acquisition method | |
| CN109582708A (en) | A kind of time series database system | |
| CN111083008A (en) | Nginx-based traffic collection and analysis method | |
| CN111125018B (en) | File exception tracing method, device, equipment and storage medium | |
| US8635333B2 (en) | Method and system for estimating a number of users of a website based on lossy compressed data | |
| CN109344163B (en) | Data verification method and device and computer readable medium | |
| CN112632568A (en) | Temperature data storage and acquisition method, system, electronic equipment and storage medium | |
| CN117170915A (en) | Data center equipment fault prediction method and device and computer equipment | |
| CN110069379B (en) | Screening method and screening device for monitoring indexes | |
| CN108121780B (en) | Data analysis model determination method and device | |
| CN115408390A (en) | Data processing method and device and electronic equipment | |
| CN109491844B (en) | Computer system for identifying abnormal information | |
| CN109978038B (en) | Cluster abnormity judgment method and device | |
| CN111327721B (en) | IP address positioning method and device, storage medium and electronic device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |