CN117472915B - Hierarchical storage method of time sequence data oriented to multiple Key values - Google Patents
Hierarchical storage method of time sequence data oriented to multiple Key values Download PDFInfo
- Publication number
- CN117472915B CN117472915B CN202311814997.XA CN202311814997A CN117472915B CN 117472915 B CN117472915 B CN 117472915B CN 202311814997 A CN202311814997 A CN 202311814997A CN 117472915 B CN117472915 B CN 117472915B
- Authority
- CN
- China
- Prior art keywords
- time
- time sequence
- key
- storage
- value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 20
- 238000013500 data storage Methods 0.000 claims abstract description 34
- 230000032683 aging Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 230000001502 supplementing effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/217—Database tuning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a hierarchical storage method of time sequence data oriented to multiple Key values, which comprises the following steps: 1. forming a Key-Value class time sequence database by time sequence data with multiple Key values under the initial set time sequence data storage time; 2. inputting judgment of access time when a user accesses the time sequence database; 3. updating the storage time, and updating the Key-Value class time sequence database according to the updated storage time; 4. time sequence data in a time period greater than the initial set time sequence data storage time is acquired from the file system binary file. The method has reasonable design, adopts a strategy that data is simultaneously stored in the time sequence database and the file system binary file, and simultaneously adjusts the storage time according to the access time of a user to the data so as to reduce the data storage capacity of the time sequence database.
Description
Technical Field
The invention belongs to the technical field of aerospace telemetry time sequence data storage, and particularly relates to a hierarchical storage method of time sequence data oriented to multiple Key values.
Background
The time sequence data is data recorded according to the same time index, the Key-Value type data is data stored according to a Key Value, and the data comprises at least one Key parameter, at least one Value parameter and a data Value. The timing data is a typical Key-Value class data in which time is a Key parameter. The Key-Value time sequence data at least comprises 2 Key parameters, namely time and other Key parameters. The timing data is typically stored in a timing database that is indexed with time and Key parameters. Sequential data access generally has obvious aging characteristics, so the sequential database supports setting sequential storage time.
For Key-Value time sequence data, key-Value time sequence data with fewer Key parameter values are generally stored in a sub-table mode, each Key Value is used as one table, data of each table becomes non-Key-Value time sequence data, the non-Key-Value time sequence data is accessed only through time as an index, and different Key values need to access different tables. For time sequence data with more Key values, thousands of sensor types and even tens of thousands of sensor types can be used as the Key values, thousands of tables need to be established, and if the instrument has n sets, the number of the tables needs to be multiplied by n times. The number of the time series database tables is set to be an upper limit, and even if part of the time series database with no upper limit exists, the data index efficiency is obviously reduced when a large number of tables are maintained. At present, a storage method for efficiently accessing Key-Value time sequence data meeting multiple Key values is lacking.
Disclosure of Invention
The technical problem to be solved by the invention is to provide a hierarchical storage method of time sequence data oriented to multiple Key values, which has simple steps and reasonable design, adopts a strategy that data is stored in a time sequence database and a file system binary file at the same time, adjusts the storage time according to the access time of a user to the data, reduces the data storage capacity of the time sequence database, and solves the problem of low time sequence data access efficiency caused by the fact that a large amount of time sequence data is completely stored in the time sequence database, and also avoids the problem that part of data cannot be accessed due to the fact that unified storage time is set.
In order to solve the technical problems, the invention adopts the following technical scheme: the hierarchical storage method of the time sequence data facing to the multiple Key values is characterized by comprising the following steps:
step one, forming a Key-Value class time sequence database by time sequence data with multiple Key values under the initial set time sequence data storage time; setting a Key Value 0, a Key Value 1, a Key Value I, a Key Value I, a Value 1, a Value J, wherein the Value values J, I and I are positive integers, I is more than or equal to 1 and is less than or equal to I, the Key Value 0 attribute is time, J and J are positive integers, and J is more than or equal to 1 and is less than or equal to J;
step two, inputting judgment of access time when a user accesses the time sequence database:
when a user accesses the Key-Value class time sequence database, inputting access time and a plurality of Key values, judging whether the access time is smaller than the initial set time sequence data storage time, and executing the third step if the access time is smaller than the initial set time sequence data storage time; if the number is greater than the preset number, executing the fourth step, and if the number is equal to the preset number, not processing;
step three, updating the storage time, and updating the Key-Value class time sequence database according to the updated storage time;
and step four, resetting the set time sequence data storage time as access time, acquiring time sequence data in a time period longer than the initial set time sequence data storage time from a file system binary file, and storing the time sequence data in the Key-Value type time sequence database in the step one to finish the updating of the Key-Value type time sequence database.
The hierarchical storage method of the time sequence data facing to the multiple Key values is characterized by comprising the following steps of: in the third step, the storage time is updated, and the specific process is as follows:
step 301, obtaining storage time T (e) after the e-th reduction according to T (e) = [ T (e-1) × (1-1/M) ]; wherein e is a positive integer greater than or equal to 1; when e=1, T (0) is an initial set time-series data storage time, M represents a time-series storage time reduction rate parameter, and M is initially a minimum integer greater than T (0); [ ] Representing rounding;
step 302, comparing the input access time with the e-th reduced storage time T (e), and if the input access time is smaller than the e-th reduced storage time T (e), executing step 303; otherwise, go to step 305;
step 303, judging whether the storage time T (e) after the e-th reduction is smaller than 1/M, and if the storage time T (e) after the e-th reduction is smaller than 1/M, updating the storage time to be zero; if greater than or equal to 1/M, execute step 304;
step 304, obtaining the e+1st reduced storage time T (e+1) according to T (e+1) = [ T (e) × (1-1/M) ], and repeating step 302;
step 305, taking the input access time as the updated storage time.
The hierarchical storage method of the time sequence data facing to the multiple Key values is characterized by comprising the following steps of: in the fourth step, when the set time sequence data storage time is reset to be the access time, the time sequence storage time reduction rate parameter M is required to be updated, and the updated time sequence storage time reduction rate parameter M is the minimum positive integer greater than the access time.
Compared with the prior art, the invention has the following advantages:
1. according to the invention, under the condition of initially setting the time sequence data storage time, the time sequence data with multiple Key values form the Key-Value type time sequence database, so that the user data request can be responded quickly by utilizing the characteristic of efficient storage of the time sequence database.
2. The invention is based on the time sequence database and the file system binary file level data storage, and solves the problems of the time sequence database such as multiple Key value parameters, capacity brought by mass data storage and the like.
3. According to the method and the device, based on the access requirement of the user, namely the access time, the storage strategy time of the Key-Value time sequence database is adjusted, and the data storage quantity is reduced, so that the efficient access of the Key-Value time sequence data is realized.
In summary, the method has simple steps and reasonable design, adopts the strategy that the data is simultaneously stored in the time sequence database and the file system binary file, adjusts the storage time according to the access time of the user to the data, and reduces the data storage amount of the time sequence database, thereby solving the problem of low time sequence data access efficiency caused by the fact that a large amount of time sequence data is completely stored in the time sequence database, and avoiding the problem that partial data cannot be accessed due to the fact that the unified storage time is set.
The technical scheme of the invention is further described in detail through the drawings and the embodiments.
Drawings
FIG. 1 is a block flow diagram of the method of the present invention.
Detailed Description
As shown in fig. 1, the hierarchical storage method for time series data of multiple Key values of the present invention includes the following steps:
step one, forming a Key-Value class time sequence database by time sequence data with multiple Key values under the initial set time sequence data storage time; setting a Key Value 0, a Key Value 1, a Key Value I, a Key Value I, a Value 1, a Value J, wherein the Value values J, I and I are positive integers, I is more than or equal to 1 and is less than or equal to I, the Key Value 0 attribute is time, J and J are positive integers, and J is more than or equal to 1 and is less than or equal to J;
step two, inputting judgment of access time when a user accesses the time sequence database:
when a user accesses the Key-Value class time sequence database, inputting access time and a plurality of Key values, judging whether the access time is smaller than the initial set time sequence data storage time, and executing the third step if the access time is smaller than the initial set time sequence data storage time; if the number is greater than the preset number, executing the fourth step, and if the number is equal to the preset number, not processing;
step three, updating the storage time, and updating the Key-Value class time sequence database according to the updated storage time;
and step four, resetting the set time sequence data storage time as access time, acquiring time sequence data in a time period longer than the initial set time sequence data storage time from a file system binary file, and storing the time sequence data in the Key-Value type time sequence database in the step one to finish the updating of the Key-Value type time sequence database.
In this embodiment, the updating of the storage time in the third step includes the following specific steps:
step 301, obtaining storage time T (e) after the e-th reduction according to T (e) = [ T (e-1) × (1-1/M) ]; wherein e is a positive integer greater than or equal to 1; when e=1, T (0) is an initial set time-series data storage time, M represents a time-series storage time reduction rate parameter, and M is initially a minimum integer greater than T (0); [ ] Representing rounding;
step 302, comparing the input access time with the e-th reduced storage time T (e), and if the input access time is smaller than the e-th reduced storage time T (e), executing step 303; otherwise, go to step 305;
step 303, judging whether the storage time T (e) after the e-th reduction is smaller than 1/M, and if the storage time T (e) after the e-th reduction is smaller than 1/M, updating the storage time to be zero; if greater than or equal to 1/M, execute step 304;
step 304, obtaining the e+1st reduced storage time T (e+1) according to T (e+1) = [ T (e) × (1-1/M) ], and repeating step 302;
step 305, taking the input access time as the updated storage time.
In the fourth embodiment, when the time-series data storage time is reset and set as the access time, the time-series storage time reduction rate parameter M needs to be updated, and the updated time-series storage time reduction rate parameter M is the smallest positive integer greater than the access time.
In this embodiment, the time-lapse storage time reduction rate parameter M is updated to facilitate adjustment according to the access time.
In this embodiment, the Key-Value class timing database sets time, key Value 1, key Value I, and Value 1, value J, value J. I represents the total number of key parameters, and J represents the total number of Value values.
In this embodiment, when the initial time-series data storage time T (0) is set to 30 days, the Key-Value class time-series database stores data within 30 days, and only the data of the last 30 days can be accessed.
In the embodiment, step three, the Key-Value class time sequence database is updated according to the updated storage time, namely, the time sequence data which is larger than the updated storage time is deleted; and step four, supplementing the time sequence data in the time period longer than the initial set time sequence data storage time into the Key-Value time sequence database, and finishing updating.
In summary, the method has simple steps and reasonable design, adopts the strategy that the data is simultaneously stored in the time sequence database and the file system binary file, adjusts the storage time according to the access time of the user to the data, and reduces the data storage amount of the time sequence database, thereby solving the problem of low time sequence data access efficiency caused by the fact that a large amount of time sequence data is completely stored in the time sequence database, and avoiding the problem that partial data cannot be accessed due to the fact that the unified storage time is set.
The foregoing description is only a preferred embodiment of the present invention, and is not intended to limit the present invention, and any simple modification, variation and equivalent structural changes made to the above embodiment according to the technical substance of the present invention still fall within the scope of the technical solution of the present invention.
Claims (2)
1. The hierarchical storage method of the time sequence data facing to the multiple Key values is characterized by comprising the following steps:
step one, forming a Key-Value class time sequence database by time sequence data with multiple Key values under the initial set time sequence data storage time; setting a Key Value 0, a Key Value 1, a Key Value I, a Key Value I, a Value 1, a Value J, wherein the Value values J, I and I are positive integers, I is more than or equal to 1 and is less than or equal to I, the Key Value 0 attribute is time, J and J are positive integers, and J is more than or equal to 1 and is less than or equal to J;
step two, inputting judgment of access time when a user accesses the time sequence database:
when a user accesses the Key-Value class time sequence database, inputting access time and a plurality of Key values, judging whether the access time is smaller than the initial set time sequence data storage time, and executing the third step if the access time is smaller than the initial set time sequence data storage time; if the number is greater than the preset number, executing the fourth step, and if the number is equal to the preset number, not processing;
step three, updating the storage time, and updating the Key-Value class time sequence database according to the updated storage time;
resetting the set time sequence data storage time as access time, acquiring time sequence data in a time period longer than the initial set time sequence data storage time from a file system binary file, and storing the time sequence data in the Key-Value type time sequence database in the step one to finish the updating of the Key-Value type time sequence database;
in the third step, the storage time is updated, and the specific process is as follows:
step 301, obtaining storage time T (e) after the e-th reduction according to T (e) = [ T (e-1) × (1-1/M) ]; wherein e is a positive integer greater than or equal to 1; when e=1, T (0) is an initial set time-series data storage time, M represents a time-series storage time reduction rate parameter, and M is initially a minimum integer greater than T (0); [ ] Representing rounding;
step 302, comparing the input access time with the e-th reduced storage time T (e), and if the input access time is smaller than the e-th reduced storage time T (e), executing step 303; otherwise, go to step 305;
step 303, judging whether the storage time T (e) after the e-th reduction is smaller than 1/M, and if the storage time T (e) after the e-th reduction is smaller than 1/M, updating the storage time to be zero; if greater than or equal to 1/M, execute step 304;
step 304, obtaining the e+1st reduced storage time T (e+1) according to T (e+1) = [ T (e) × (1-1/M) ], and repeating step 302;
step 305, taking the input access time as the updated storage time.
2. The hierarchical storage method for time series data facing multiple Key values according to claim 1, wherein the method comprises the following steps: in the fourth step, when the set time sequence data storage time is reset to be the access time, the time sequence storage time reduction rate parameter M is required to be updated, and the updated time sequence storage time reduction rate parameter M is the minimum positive integer greater than the access time.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311814997.XA CN117472915B (en) | 2023-12-27 | 2023-12-27 | Hierarchical storage method of time sequence data oriented to multiple Key values |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311814997.XA CN117472915B (en) | 2023-12-27 | 2023-12-27 | Hierarchical storage method of time sequence data oriented to multiple Key values |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117472915A CN117472915A (en) | 2024-01-30 |
| CN117472915B true CN117472915B (en) | 2024-03-15 |
Family
ID=89635130
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311814997.XA Active CN117472915B (en) | 2023-12-27 | 2023-12-27 | Hierarchical storage method of time sequence data oriented to multiple Key values |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117472915B (en) |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102725755A (en) * | 2011-12-31 | 2012-10-10 | 华为技术有限公司 | Method and system of file access |
| CN106446081A (en) * | 2016-09-09 | 2017-02-22 | 西安交通大学 | Method for mining association relationship of time series data based on change consistency |
| CN106446278A (en) * | 2016-10-24 | 2017-02-22 | 北京亚控科技发展有限公司 | Method for searching data target on basis of spatial-temporal database |
| CN106776823A (en) * | 2016-11-25 | 2017-05-31 | 华为技术有限公司 | A kind of time series data management method, equipment and device |
| CN107092628A (en) * | 2017-01-10 | 2017-08-25 | 口碑控股有限公司 | The treating method and apparatus of time series data |
| CN107665255A (en) * | 2017-09-30 | 2018-02-06 | 杭州时趣信息技术有限公司 | Method, apparatus, equipment and the storage medium of key value database data change |
| CN109189863A (en) * | 2016-10-24 | 2019-01-11 | 北京亚控科技发展有限公司 | A method of description things time attribute is simultaneously searched based on the description |
| CN110046183A (en) * | 2019-04-16 | 2019-07-23 | 北京易沃特科技有限公司 | A kind of time series data polymerization search method, equipment and medium |
| CN110109923A (en) * | 2019-04-04 | 2019-08-09 | 北京市天元网络技术股份有限公司 | Storage method, analysis method and the device of time series data |
| CN111078753A (en) * | 2019-12-17 | 2020-04-28 | 联想(北京)有限公司 | HBase database-based time sequence data storage method and device |
| CN114048238A (en) * | 2022-01-12 | 2022-02-15 | 树根互联股份有限公司 | Storage method and device for industrial equipment time sequence data and electronic equipment |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP6029951B2 (en) * | 2012-11-27 | 2016-11-24 | 株式会社日立製作所 | Time series database setting automatic generation method, setting automatic generation system and monitoring server |
-
2023
- 2023-12-27 CN CN202311814997.XA patent/CN117472915B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102725755A (en) * | 2011-12-31 | 2012-10-10 | 华为技术有限公司 | Method and system of file access |
| CN106446081A (en) * | 2016-09-09 | 2017-02-22 | 西安交通大学 | Method for mining association relationship of time series data based on change consistency |
| CN106446278A (en) * | 2016-10-24 | 2017-02-22 | 北京亚控科技发展有限公司 | Method for searching data target on basis of spatial-temporal database |
| CN109189863A (en) * | 2016-10-24 | 2019-01-11 | 北京亚控科技发展有限公司 | A method of description things time attribute is simultaneously searched based on the description |
| CN106776823A (en) * | 2016-11-25 | 2017-05-31 | 华为技术有限公司 | A kind of time series data management method, equipment and device |
| CN107092628A (en) * | 2017-01-10 | 2017-08-25 | 口碑控股有限公司 | The treating method and apparatus of time series data |
| CN107665255A (en) * | 2017-09-30 | 2018-02-06 | 杭州时趣信息技术有限公司 | Method, apparatus, equipment and the storage medium of key value database data change |
| CN110109923A (en) * | 2019-04-04 | 2019-08-09 | 北京市天元网络技术股份有限公司 | Storage method, analysis method and the device of time series data |
| CN110046183A (en) * | 2019-04-16 | 2019-07-23 | 北京易沃特科技有限公司 | A kind of time series data polymerization search method, equipment and medium |
| CN111078753A (en) * | 2019-12-17 | 2020-04-28 | 联想(北京)有限公司 | HBase database-based time sequence data storage method and device |
| CN114048238A (en) * | 2022-01-12 | 2022-02-15 | 树根互联股份有限公司 | Storage method and device for industrial equipment time sequence data and electronic equipment |
Non-Patent Citations (1)
| Title |
|---|
| 基于Hadoop的工业大数据存储分析系统;范旭辉;;科技创新与应用;20200731(第23期);第 24-26页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117472915A (en) | 2024-01-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103942342B (en) | Memory database OLTP and OLAP concurrency query optimization method | |
| CN105320775B (en) | The access method and device of data | |
| CN108256088A (en) | A kind of storage method and system of the time series data based on key value database | |
| CN110209728A (en) | A kind of Distributed Heterogeneous Database synchronous method, electronic equipment and storage medium | |
| CN110222029A (en) | A kind of big data multidimensional analysis computational efficiency method for improving and system | |
| CN107463610B (en) | Data warehousing method and device | |
| CN107977446A (en) | A kind of memory grid data load method based on data partition | |
| JP7375173B2 (en) | Data processing method and related equipment, and computer program | |
| CN104572809B (en) | A kind of distributed relational database spread method | |
| US20230067182A1 (en) | Data Processing Device and Method, and Computer Readable Storage Medium | |
| CN112632068A (en) | Solution for rapidly providing mass data query service | |
| CN104408128B (en) | A kind of reading optimization method indexed based on B+ trees asynchronous refresh | |
| CN117472915B (en) | Hierarchical storage method of time sequence data oriented to multiple Key values | |
| CN103605732A (en) | Data warehouse, data warehouse system and data warehouse construction method based on Infobright | |
| CN117235028A (en) | Data query method and device based on log file | |
| CN112269802A (en) | Method and system for frequent deletion, modification and check optimization based on Clickhouse | |
| CN104268146A (en) | Static B+-tree index method suitable for analytic applications | |
| US8676769B2 (en) | Method for defragmenting indexes in a relational database and system for performing the method | |
| US9830323B2 (en) | Method and system for archiving data from a source database to a target database | |
| CN114490724B (en) | Method and device for processing database query statement | |
| CN116431635A (en) | Lake and warehouse integrated-based power distribution Internet of things data real-time processing system and method | |
| CN115576924A (en) | Data migration method | |
| CN112269797B (en) | Multidimensional query method of satellite remote sensing data on heterogeneous computing platform | |
| CN103617181A (en) | Method and device for establishing universal database of relationships | |
| Zhu et al. | Developing a dynamic materialized view index for efficiently discovering usable views for progressive queries |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |