CN104090987A - Historical data storage and indexing method - Google Patents
Historical data storage and indexing method Download PDFInfo
- Publication number
- CN104090987A CN104090987A CN201410364895.7A CN201410364895A CN104090987A CN 104090987 A CN104090987 A CN 104090987A CN 201410364895 A CN201410364895 A CN 201410364895A CN 104090987 A CN104090987 A CN 104090987A
- Authority
- CN
- China
- Prior art keywords
- data
- interim
- file
- archive file
- archives
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2477—Temporal data queries
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a historical data storage and indexing method and belongs to the field of real-time historical databases. According to the characteristics of industrial historical data storage formats, a simple and efficient historical data storage method and an efficient indexing mechanism are given. According to the historical data storage method, on the one hand, the I/O of a magnetic disk is reduced as much as possible by performing compression and cache on historical data and optimizing the data archiving sorting process; on the other hand, the I/O load of the magnetic disk in the data archiving sorting process is distributed to all time points in a balanced mode, and thus it is guaranteed that the requirement for storage of current historical data is met, and meanwhile the storage method has the dynamic extension characteristic. Through the indexing mechanism, the historical data can be fast accessed, and when a database is dynamically extended, the system can be not influenced by data access by dynamically modifying indexing files.
Description
Technical field
The invention belongs to real time historical database technical field, more specifically, relate to a kind of history data store and indexing means.
Background technology
Industrial informationization has become the theme of 21 century whole world industrial circle, and IT application process started to deepen continuously with a kind of impetus of depth & wideth common development in recent years.Along with the rise of Internet of Things, large quantity sensor and intelligent terminal are widely used in traditional industry field, have further promoted the development process of industrial information.
In traditional commercial production flow process, real-time data base is widely used.It is mainly used in real-time monitoring and the data persistence of the industrial production system that some data volumes are less, and the industry relating to comprises electric power, oil, metallurgy, chemical industry etc.Real time historical database is usingd file data point and is described production data as base unit.A physical data source in the file data point respective sensor of a database, such as the voltage of a certain power transmission line, the temperature of some check points etc.The data that file data point collects are discrete datas, all with timestamp.From collection in worksite to production process data through being finally kept at after a series of processing in the middle of real time historical database.These type of data conventionally follow-up process recall or the process of data analysis in use.
Along with the deep development of industrial information, decision maker is more and more higher to the requirement of the storage efficiency of historical data and data access speed, and the problem of the historical data access efficiency deficiency that therefore current real time historical database exists is more outstanding.In the face of historical data access efficiency requires more and more higher demand, there is no at present complete available embodiment and both can meet current data access requirement, the system that guarantees again has the demand that dynamic scalability is dealt with the data Storage and Processing of sustainable growth.
Summary of the invention
Above defect or Improvement requirement for prior art, the invention provides a kind of storage and indexing means of historical data, according to the feature of industrial data storage format, a kind of simple storage means and a kind of efficient Indexing Mechanism of history data file have efficiently been provided, can meet the efficient access requirement of current historical data, can guarantee to have the characteristic of dynamic expansion simultaneously.This storage means arranges flow process by compressed cache historical data and optimization data filing on the one hand magnetic disc i/o is reduced as much as possible, on the other hand by being distributed on each time point the magnetic disc i/o load balancing in filing arrangement process, thereby the memory requirement that guarantees to meet current historical data, this storage means has the characteristic of dynamic expansion simultaneously.This Indexing Mechanism can be realized the fast access to historical data, when database carries out dynamic expanding simultaneously, can make system unaffected for the access of data by on-the-fly modifying index file.
For achieving the above object, according to one aspect of the present invention, provide a kind of storage means of historical data, having comprised:
Step 1 reads a historical data from level cache and inserts request, the numbering that described insertion request comprises archives point, the data value of described archives point and the timestamp of described data value, and described archives point refers to the time series data of a data source;
Step 2 utilizes Lossy Compression Algorithm to carry out lossy compression method to the described data value in described insertion request, if described data value does not have compressedly to fall, described data value and described timestamp are write described archives point L2 cache corresponding in writing the level 2 buffering district of state, in described L2 cache, the equal corresponding Liang Ge level 2 buffering of each archives point district, one of them level 2 buffering district is when writing state, another is just in read states, when the described data value of an archives point and described timestamp write described L2 cache, first determine the current in writing the level 2 buffering district of state of this archives point correspondence, then according to the buffer zone head in level 2 buffering district, determine concrete writing position, complete again writing of described data value and described timestamp, final updating is current in writing the buffer zone header in the level 2 buffering district of state,
If step 3 is current, in writing, the level 2 buffering district of state has been write completely or its timer regularly arrives, the reading and writing state in the described Liang Ge level 2 buffering district of this archives point correspondence is exchanged, and the filing historical data that the level 2 buffering district in read states is stored writes interim archive file, described interim archive file is only stored the filing historical data of today, concrete writing position is inquired about at interim filing index file according to archives point numbering, after completing and writing, again upgrade the writing position information of interim filing index file;
Step 4 is whenever entering new one day, traversal all files point, the filing historical data by it in interim archive file all writes formal archive file, always writes the afterbody of described formal archive file at every turn, after completing and writing, then will write starting position and be recorded to filing index file; Before arranging interim archive file, be first within new one day, to create interim filing index file and interim archive file; Arrange complete after, delete the interim filing index file arranged and archive file temporarily.
According to another aspect of the present invention, a kind of indexing means of historical data is provided, comprising:
Step 1 is obtained an inquiry request, the initial time T that it comprises archives point numbering, inquiry
bEGINwith end time T
eND;
Step 2 is divided into N the subquery task in same nature sky by the query time section comprising in described inquiry request by nature sky, and wherein N represents the nature day number that described query time section comprises;
Step 3 is obtained a unenforced subquery task;
Step 4, according to " year-month-day " information of the initial time of described subquery task, judges whether initial time is today, if it is performs step 5, otherwise execution step 7;
Step 5 is calculated the offset address of interim indexing units correspondence in the interim index file of today of archives point according to the archives point numbering in described subquery task, computing formula is as follows:
Interim indexing units offset address=sizeof (interim indexing units) * (archives point numbering-1)
Wherein, sizeof (interim indexing units) represents the length of the interim indexing units of each archives point; According to the offset address of described interim indexing units, from the interim index file of today, read the interim indexing units of described archives point, thereby obtain first address and the current data block first address of first data block;
Step 6 is from the first data block of described archives point, travel through whole data block chained list, data field in read block be cached to compressed data set Data in order, also read the packed data in the current level 2 buffering district in writing state that described archives point is corresponding and be cached to compressed data set Data, continuing execution step 9;
The index block of " year-month-day " acquisition of information date index file of the initial time that step 7 comprises according to described subquery task, further from described index block, obtain formal archive file stores position and filename, corresponding filing index file storage directory and filename, and date index data item corresponding to Query Dates, further according to the filing index data in described date index data item, offset address and the archives point in correspondence filing index file numbered, obtain the index data of this day described archives point of Query Dates, offset address and the length of the data that comprise described Query Dates this day in the formal archive file of correspondence,
The index information that step 8 obtains according to described step 7 reads out described archives and puts this day corresponding compressed data set Data from corresponding formal archive file;
Step 9, according to the Data Update cycle of described archives point, decompresses to the data in described compressed data set Data, obtains in chronological sequence tactic described archives and puts daylong decompressed data set DeData;
If the initial time of the described subquery task of step 10 equals the end time, represent the data of a special time stamp of inquiry, in described decompressed data collection DeData, search the data D that timestamp equals initial time
aif there is return data D
aotherwise return to empty result set, if described initial time is not equal to the described end time, the data that represent a special time period of inquiry, by in described decompressed data collection DeData, meet data that timestamp is more than or equal to described initial time and is less than or equal to described end time condition in chronological sequence order be inserted into the set of subquery task result, by i sub-query task T
iinquire about resulting result data collection and be designated as RS
i, wherein, the span of i is 1～N;
Step 11 judges whether that the equal executed of all subquery tasks is complete, if not continuing to carry out described step 3, otherwise execution step 12;
Step 12 is by the resulting result data collection of all subquery tasks RS
iaccording to time order and function order, merge, obtain archives point to be checked at described initial time T
bEGINto described end time T
eNDall historical datas in time period, finally return to the net result collection after merging.
In general, the above technical scheme of conceiving by the present invention compared with prior art, has following beneficial effect:
First, utilize being used alternatingly of A, two groups of L2 caches of B, can improve the buffer memory ability of data, prevent loss of data;
Secondly, by designing interim archive file and interim index file, in view of its file structure simple, the disk operating number of times in the time of can greatly reducing data filing, thus improve the filing ability of data;
Again, by designing interim archive file method for sorting and the formal archive file of deflation formula, can greatly reduce the storage space of formal archive file;
Finally, by date index and filing index, can greatly be reduced in the disk operating number of times in query script in formal archive file, thereby improve search efficiency.
Accompanying drawing explanation
Fig. 1 is the general illustration of historical data persistent storage process in the embodiment of the present invention;
Fig. 2 is the structural representation of embodiment of the present invention L2 cache;
Fig. 3 is the interim archive file and interim indexed file structure schematic diagram in the embodiment of the present invention;
Fig. 4 is the structural representation of formal archive file in the embodiment of the present invention, date index file and filing index file;
Fig. 5 is data persistence storage means process flow diagram in the embodiment of the present invention;
Fig. 6 stores the interim filing of the data in L2 cache into method flow diagram in interim archive file in the embodiment of the present invention;
Fig. 7 in the embodiment of the present invention arranges interim archive file the method flow diagram of formal archive file;
Archives point of index that Fig. 8 provides for the embodiment of the present invention is the method flow diagram of historical data in section at any time.
Embodiment
In order to make object of the present invention, technical scheme and advantage clearer, below in conjunction with drawings and Examples, the present invention is further elaborated.Should be appreciated that specific embodiment described herein, only in order to explain the present invention, is not intended to limit the present invention.In addition,, in each embodiment of described the present invention, involved technical characterictic just can not combine mutually as long as do not form each other conflict.
The requirement of storing for precision due to industrial historical data is not very high, therefore there is no need all data all to store, and need to compress it.The number of times of writing disk when reducing historical data persistent storage, is necessary first historical data to be first cached, thereby can the disposable historical data by a period of time writes in data in magnetic disk file.
Figure 1 shows that the general illustration of historical data persistent storage process in the embodiment of the present invention.In embodiments of the present invention, historical data has mainly experienced from being cached to interim archive file, then the process from interim archive file to formal archive file.The first initialization of system, comprises and creates level cache, L2 cache and Miscellaneous Documents.Wherein, level cache adopts the structure of fifo queue, and its large I is set according to archives point quantity and renewal speed thereof, and computing formula is as follows:
Cache1Size=Num (archives point) * sizeof (data are inserted request)
Wherein, the archives point transformation comprising in Num (archives point) expression system, this upper limit is limited to the processing power of the hardware platform of the present invention's operation; Sizeof (data are inserted request) represents a required storage space of data insertion request, and preferred parameter is 64 bytes.The structure of L2 cache and large young pathbreaker are elaborated being below combined Fig. 2.The establishment of Miscellaneous Documents will below be elaborated in conjunction with Fig. 5.
External data source module inserts RQ cycle by the data that comprise archives point and writes level cache, wherein, archives point refers to the time series data of a data source, and it is numbered since 1, and the data of each archives point can be floating number, shaping number or Boolean type number.External data source module can insert request to level cache data writing continuously, in the embodiment of the present invention system also can be synchronously from level cache constantly, reading out data inserts request one by one, again the data in the request of inserting are carried out to lossy compression method, what in the embodiment of the present invention, adopt is swinging door compression algorithm, but be not limited to this algorithm, be equivalent to only deposit flex point data or some accidental datas in L2 cache.Data that can buffer memory a period of time in L2 cache, when L2 cache timer arrives or L2 cache is write full time, system will write the data in L2 cache in interim archive file and go.Interim archive file is only stored the data of today, after wait until and having crossed today, can be in the time-division in morning of second day, and after the data of guaranteeing the previous day have all write temporary file, system can write formal archive file by temporary file arrangement.
Figure 2 shows that the structural representation of embodiment of the present invention L2 cache.L2 cache is divided into the buffer zone of a plurality of archives points, each archives point has A, B Liang Zu level 2 buffering district, Gai Liangge level 2 buffering district always keeps one for read states, another is for writing state, when the cycle of the filing timer in one of them level 2 buffering district arrives Huo Gai level 2 buffering district and writes when full, exchange the read-write state in Liang Ge level 2 buffering district.Each level 2 buffering district includes buffer zone head and data, and buffer zone head comprises the timestamp of first data, the timestamp of latest data and the size that taken up room in read-write state zone bit, level 2 buffering district.Wherein, read-write state zone bit is for identifying the current state in this level 2 buffering district; The timestamp of first data represents the timestamp of first data of preserving in this level 2 buffering district, is accurate to millisecond, and each timestamp occupies 9 bytes, and its middle age occupies 2 bytes, the moon, day, time, minute, second occupies respectively 1 byte, millisecond occupies 2 bytes; The timestamp of latest data represents the timestamp (even if the compressed compression algorithm of these data is fallen) of latest data; The size of having stored data content in the current level 2 buffering district that taken up room large small records.The packet in level 2 buffering district is containing timestamp and the data value of map file point.
The size in each group level 2 buffering district of each archives point is calculated according to following formula:
Cache2Size=Items * [sizeof (data value)+sizeof (timestamp)]
Wherein, Items represents the data item number of system maximum storage, and preferred parameter is 256; Sizeof (data value) represents the required storage space of each data value, by data type and the operating system addressing bit number of data value, is jointly determined, operating system addressing bit number is generally 32 or 64; Sizeof (timestamp) represents the required storage space of timestamp, timestamp comprise " during year-month-day: minute: second. millisecond ", be 9 bytes in embodiments of the present invention.
Figure 3 shows that the interim archive file of the embodiment of the present invention and the structural representation of interim index file.Wherein interim archive file name shape is as tData_YYYY-MM-DD.dat, for example tData_2014-3-19.dat; Interim index file name shape is as tIndex_YYYY-MM-DD.index, for example tIndex_2014-3-19.index.Interim archive file include file Tou He filing data district, file header comprises: state flag bit and file size, state flag bit illustrates that this interim archive file is current in what state, and in embodiments of the present invention, concrete state comprises free time, filing and arrangement etc.; File size comprises the size of filing data district and file header.As shown in Figure 3, in interim index file, for each archives point has distributed an interim indexing units, in interim indexing units, storing first address and the current data block first address that these archives are put first data block.Wherein, the first address of first data block refers to the offset address of first data block in interim archive file; Current data block first address refer to current just the data block of filing data can be for the region of data writing the offset address in interim archive file.The filing data district of interim archive file is comprised of a plurality of archives point data blocks, the Historical archiving data in the corresponding archives point a period of time section of each archives point data block, and each archives point can comprise a plurality of archives point data blocks.Archives point data block comprises three parts: data field size, Next pointer and data field.Wherein, the data of data area stores archives point, data field size is the shared storage size of data in data field; Next pointer is next archives point data block offset address in interim archive file, thereby forms an archives point data block chained list.
System can be carried out the memory file point historical data of a day for each archives point predistribution one blocks of data piece in interim archive file initialized time, and now data field size is initialized as 0, Next pointer and is initialized as address blank.Meanwhile, the first address of the first data block of each archives point in interim indexing units and last data block first address are all initialized as predistribution data block at the offset address in interim archive file.
Data in L2 cache are written into interim archive file, need according to the byte number of each writing data blocks more new data block write offset address.And if current data block has been fully written, system can be distributed at interim archive file end a data block for this archives point appends again, and with the new offset address of distribute data piece in archive file temporarily under the Next pointer record of current this data block.Now, the first address of the first data block in interim indexing units remains unchanged, and current data block first address will be updated to the offset address of new distribute data piece in interim archive file.
Interim archive file and interim index file all belong to temporary file, after data preparation in the interim archive file of yesterday, the temporary file of yesterday all can be deleted, also can create the temporary file of tomorrow simultaneously, make system always have interim archive file and the interim index file of today and tomorrow, so that the temporary file of every day can be accomplished seamless switching, use.
Figure 4 shows that the structural representation of formal archive file in the embodiment of the present invention, date index file and filing index file, and the relation between these files.Wherein formal archive file name shape is as hisdata_N.dat, N representation file sequence number wherein, and for example first formal archive file is hisdata_1.dat; Date index file has and only has one, and its filename shape is as time.index; Filing index file is corresponding with formal archive file, and its filename shape is as hisdata_N.index, and wherein N is consistent with the sequence number in formal archive file name.For guaranteeing that the present invention is applicable to different file, the storage cap of each formal archive file all can arrange according to actual file system, for example: in FAT32 file system, the storage cap of formal archive file can not surpass 4GB, new technology file system storage cap can be made as the value that is greater than 4GB, the concrete upper limit can also be modified according to practical application request, but the preferred formal archive file stores upper limit should be lower than 2GB.When a formal archive file reaches the storage cap of setting, create a new formal archive file and filing index file, its sequence number all will increase 1 certainly.In embodiments of the present invention, formal archive file as shown in Figure 4, starts each archives point of sequential storage data of every day from file header.
Date index file consists of a plurality of index blocks, the corresponding formal archive file of each index block.Each index block all comprises Start Date (comprising year, month, day), Close Date (comprising year, month, day), the number of days of storage data, formal archive file stores catalogue and the filename of the data of storing in corresponding formal archive file, formal corresponding filing index file storage directory and the filename of archive file, and date index data item.Wherein, the number of days of the quantity of date index data item and the data of storage equates.Each date index data item comprises date and the offset address of filing index data in correspondence filing index file thereof.
The corresponding formal archive file of filing index file, the number of days of storing data in formal archive file equals to file the number of days of filing index data in index file.Filing index file consists of a plurality of filing index data pieces, the index information of all files point data of a day in the corresponding formal archive file of each filing index data piece.In filing index data piece, the index data of each archives point has comprised offset address and the data length of this archives point data of a day in the formal archive file of correspondence.
The process flow diagram that Figure 5 shows that data persistence storage means in the embodiment of the present invention, specifically comprises the following steps:
In step 501, system initialization, creates level cache, and by mode mentioned above, is one group of L2 cache of each archives point establishment according to the maximum archives amount of counting in system;
In step 502, whether inspection there is the interim archive file before today or today, if it is performs step 503, otherwise execution step 504;
In step 503, the interim archive file sorting and file existing is arrived to formal archive file, and after completing, delete this interim archive file and interim index file;
In step 504, check whether there is date index file, if there is no according to the newly-built date index file of the naming rule of aforementioned date index file; Check and whether have formal archive file, if there is no according to aforementioned formal archive file and the newly-built empty formal archive file of naming rule of filing index file and corresponding filing index file thereof, and insert corresponding index block at the afterbody of date index file; According to the interim archive file of naming rule newly-built today of difference of aforementioned interim archive file and interim index file and tomorrow and interim index file;
In step 505, from level cache, read historical data and insert request, archives point numbering and timestamp corresponding to data in analysis request;
In step 506, if the data time inserting in request stabs early than the current latest data timestamp of preserving of this archives point, abandon this insertion request in writing the L2 cache of state, re-execute step 505, otherwise execution step 507;
In step 507, the timestamp of the data that are this by the latest data update of time stamp in the buffer zone head in level 2 buffering district in writing state corresponding to archives point, and carry out Lossy Compression Algorithm, swinging door compression algorithm for example, but be not limited to this algorithm;
In step 508, judging whether data are compressed falls, and is to perform step 509, otherwise execution step 510;
In step 509, according to the current time in system, judge whether the filing cycle arrive, this filing cycle can be determined by application request, the embodiment of the present invention preferably the filing cycle be 10 minutes, if it is perform step 512, otherwise execution step 505;
In step 510, by data and timestamp thereof write archives point corresponding in writing the level 2 buffering district of state, and upgrade the latest data timestamp in the head of buffer zone, and with taking up room, add that the byte number renewal of this data writing takes up room;
In step 511, according to taking up room in the buffer zone head in level 2 buffering district in writing state corresponding to archives point, judge whether this level 2 buffering district has write full, or judge according to the current time in system whether the filing cycle arrives, if it is perform step 512, otherwise execution step 505;
In step 512, the A of archives point, B Liang Zu level 2 buffering district read-write state are exchanged, and the data that the level 2 buffering district that is replaced by read states is stored write interim archive file and reinitialize this level 2 buffering district;
In step 513, check system current time, judges whether to enter new one day, if it is performs step 514, otherwise execution step 505;
In step 514, the interim archive file sorting and file of yesterday is arrived to formal archive file, after completing, delete interim archive file and the interim index file of yesterday, and create interim archive file and the interim index file of tomorrow, so that the temporary file of every day can be accomplished seamless switching, use.
Figure 6 shows that in the embodiment of the present invention that by the archives point data conversion storage in L2 cache, to the process flow diagram in interim archive file, specifically comprises the following steps:
In step 601, according to the state flag bit of the interim archive file of today, judge that interim archive file, whether in idle condition, if it is performs step 602, otherwise execution step 603;
In step 602, the state flag bit of the interim archive file of today is updated to write state;
In step 603, obtain an archives point numbering, and according to archives point numbering, from interim index file, read the interim indexing units of this archives point, utilize the current data block first address in interim indexing units, from interim archive file, read the data field size of current data block, and according to archives point numbering, obtain the address information of its L2 cache;
In step 604, according to taking up room in the buffer zone head in the level 2 buffering district in read states corresponding to this archives point, the data field size of current data block and the upper limit of data block, can judgement current data block hold all data in this level 2 buffering district in read states, if can, perform step 606, otherwise execution step 605;
In step 605, in a data block of the new establishment of afterbody of interim archive file, preferred data block size is 16K byte; Offset address by the new data block creating of the Next pointed of current data block in interim archive file, and the current data block first address in the interim indexing units in interim index file is updated to the offset address of new establishment data block in interim archive file by this archives point;
In step 606, the data in the level 2 buffering district by this in read states write current data block, and taking up room in the buffer zone head in the level 2 buffering district with this in read states upgraded the data field size of current data block; The timestamp of first data in level 2 buffering district by this in read states,, the timestamp of latest data and the size that taken up room are all initialized as 0, continues execution step 603 until all archives points are all finished dealing with.
Figure 7 shows that in the embodiment of the present invention and arrange the method flow diagram that interim archive file forms formal archive file, specifically comprise the following steps:
In step 701, whether system Real-Time Monitoring current time is to interim archive file data preparation is write to the time threshold T of formal archive file
orderif arrived time threshold T
orderperform step 702, otherwise continue Real-Time Monitoring current time, in embodiments of the present invention, preferred time threshold T
orderit is 15 minutes, to guarantee that the data in all L2 caches are all written into the interim archive file of yesterday;
In step 702, the packed data capacity sum of all files point in the interim archive file of scanning yesterday, judges according to formal archive file maxsize whether current formal archive file can hold, and is to perform step 705, otherwise performs step 703;
In step 703, according to archives point numbering order from small to large, arrange the data in the interim archive file of yesterday, the data of each archives point corresponding data piece in interim archive file are integrated together according to time sequencing, obtain this archives point compressed data set of a day;
In step 704, according to archives point numbering order from small to large, the compressed data set of each archives point is written to current formal archive file one by one, simultaneously, in corresponding filing index file, increase a filing index data piece, offset address and the length thereof of the compressed data set that records each archives point in current formal archive file, then, at date index file afterbody, increase a date index data item, offset address and the corresponding date thereof of the filing index data piece that record newly increases in current filing index file;
In step 705, according to the newly-built empty formal archive file of naming rule and the corresponding filing index file thereof of aforementioned formal archive file and filing index file, and insert corresponding index block at the afterbody of date index file, using the formal archive file newly creating as current formal archive file, by data set write newly-built archive file and update date index file and filing index file;
In step 706, when the interim archive file of yesterday arrange complete after, the interim archive file of yesterday and interim index file can be deleted, and the interim archive file of newly-built tomorrow and interim index file are to guarantee that every day, temporary file was accomplished seamless switching in across day data storage.
Because historical data same archives point packed data on the same day in formal archive file exists together, therefore the indexing means of the embodiment of the present invention can be located the historical data in certain time period in an archives point one day fast, also can the concrete historical data constantly of index.
Figure 8 shows that archives point of index that the embodiment of the present invention provides method flow diagram of historical data in section at any time, specifically comprise the following steps:
In step 801, obtain an inquiry request, the initial time T that it comprises archives point numbering, inquiry
bEGINwith end time T
eND, initial time T
bEGINwith end time T
eNDall comprise " during year-month-day: minute: second. millisecond ";
In step 802, the query time section comprising in inquiry request is divided into N the subquery task in same nature sky by nature sky, the nature day number that the query time section that wherein N represents to comprise in inquiry request comprises, the beginning and ending time of each subquery task is in same nature day; Use T
irepresent i sub-query task, the span of i is 1～N, wherein subquery task T
1beginning and ending time be initial time T
bEGINto T
bEGIN999 milliseconds of 23: 59: 59 of nature sky, place, subquery task T
nbeginning and ending time be end time T
eNDplace nature day 0: 0: 00 millisecond to T
eND, and subquery task T
2, T
3..., T
n-1represent that tundish contains a complete time in nature sky, 0: 0: 0 in same nature sky is 0 millisecond to 23: 59: 59 999 milliseconds;
In step 803, obtain a unenforced subquery task;
In step 804, according to " year-month-day " information of the initial time of subquery task, judge whether the initial time of this subquery task is today, if it is perform step 805, otherwise execution step 807;
In step 805, according to the archives point numbering in subquery task, calculate the offset address of interim indexing units correspondence in the interim index file of today of this archives point, computing formula is as follows:
Interim indexing units offset address=sizeof (interim indexing units) * (archives point numbering-1)
Wherein, sizeof (interim indexing units) represents the length of the interim indexing units of each archives point, and preferred parameter is 16 bytes in embodiments of the present invention; According to the offset address of interim indexing units, from the interim index file of today, read the interim indexing units of this archives point, thereby obtain first address and the current data block first address of first data block;
In step 806, from the first data block of this archives point, travel through whole data block chained list, data field in read block be cached to compressed data set Data in order, also read the packed data in the current level 2 buffering district in writing state that this archives point is corresponding and be cached to compressed data set Data, continuing execution step 809;
In step 807, the index block of " year-month-day " acquisition of information date index file of the initial time comprising according to subquery task, further from index block, obtain formal archive file stores position and filename, corresponding filing index file storage directory and filename, and date index data item corresponding to Query Dates, further according to the filing index data in date index data item, offset address and the archives point in correspondence filing index file numbered, obtain the index data of Query Dates this archives point of this day, offset address and the length of the data that comprise Query Dates this day in the formal archive file of correspondence,
In step 808, the index information obtaining according to step 807 reads out these archives and puts this day corresponding compressed data set Data from corresponding formal archive file;
In step 809, according to the Data Update cycle of this archives point, the data in compressed data set Data are decompressed, obtain in chronological sequence tactic these archives and put daylong decompressed data set DeData;
In step 810, if the initial time of subquery task equals the end time, represent the data of a special time stamp of inquiry, in decompressed data collection DeData, search the data D that timestamp equals initial time
aif there is return data D
a, otherwise return to empty result set; If initial time is not equal to the end time, the data that represent a special time period of inquiry, by decompressed data collection DeData, meet data that timestamp is more than or equal to initial time and is less than or equal to end time condition in chronological sequence order be inserted into the set of subquery task result; By i sub-query task T
iinquire about resulting result data collection and be designated as RS
i;
In step 811, judge whether that the equal executed of all subquery tasks is complete, if not continuing execution step 803, otherwise execution step 812;
In step 812, within N time period, the resulting result data collection of all subquery tasks RS
iaccording to time order and function order, merge, obtain archives point to be checked at initial time T
bEGINto end time T
eNDall historical datas in time period, finally return to the net result collection after merging.
Those skilled in the art will readily understand; the foregoing is only preferred embodiment of the present invention; not in order to limit the present invention, all any modifications of doing within the spirit and principles in the present invention, be equal to and replace and improvement etc., within all should being included in protection scope of the present invention.
Claims (9)
1. a historical data storage method, is characterized in that, comprising:
Step 1 reads a historical data from level cache and inserts request, the numbering that described insertion request comprises archives point, the data value of described archives point and the timestamp of described data value, and described archives point refers to the time series data of a data source;
Step 2 utilizes Lossy Compression Algorithm to carry out lossy compression method to the described data value in described insertion request, if described data value does not have compressedly to fall, described data value and described timestamp are write described archives point L2 cache corresponding in writing the level 2 buffering district of state, in described L2 cache, the equal corresponding Liang Ge level 2 buffering of each archives point district, one of them level 2 buffering district is when writing state, another is just in read states, when the described data value of an archives point and described timestamp write described L2 cache, first determine the current in writing the level 2 buffering district of state of this archives point correspondence, then according to the buffer zone head in level 2 buffering district, determine concrete writing position, complete again writing of described data value and described timestamp, final updating is current in writing the buffer zone header in the level 2 buffering district of state,
If step 3 is current, in writing, the level 2 buffering district of state has been write completely or its timer regularly arrives, the reading and writing state in the described Liang Ge level 2 buffering district of this archives point correspondence is exchanged, and the filing historical data that the level 2 buffering district in read states is stored writes interim archive file, described interim archive file is only stored the filing historical data of today, concrete writing position is inquired about at interim filing index file according to archives point numbering, after completing and writing, again upgrade the writing position information of interim filing index file;
Step 4 is whenever entering new one day, traversal all files point, the filing historical data by it in interim archive file all writes formal archive file, always writes the afterbody of described formal archive file at every turn, after completing and writing, then will write starting position and be recorded to filing index file; Before arranging interim archive file, be first within new one day, to create interim filing index file and interim archive file; Arrange complete after, delete the interim filing index file arranged and archive file temporarily.
2. the method for claim 1, is characterized in that, before described step 1, also comprises initialization operation, and described initialization operation specifically comprises following sub-step:
(0-1) create described level cache and described L2 cache, and for each archives point create one group described in level 2 buffering district, its size is calculated according to following formula:
Cache2Size=Items * [sizeof (data value)+sizeof (timestamp)]
Wherein, Items represents the data item number of system maximum storage; Sizeof (data value) represents the required storage space of each data value; Sizeof (timestamp) represents the required storage space of each timestamp;
(0-2) check the interim archive file whether exist before today or today, if it is execution step (0-3), otherwise execution step (0-4);
(0-3) the interim archive file sorting and file existing is arrived to formal archive file, and after completing, delete described interim archive file and interim index file;
(0-4) check whether there is date index file, if there is no a newly-built date index file; Check whether there is formal archive file, if there is no a newly-built empty formal archive file and corresponding filing index file thereof, and insert corresponding index block at the afterbody of described date index file; The interim archive file of newly-built today and tomorrow and temporarily index file respectively.
3. method as claimed in claim 1 or 2, it is characterized in that, each level 2 buffering district includes buffer zone head and data, described buffer zone head comprises the timestamp of first data, the timestamp of latest data and the size that taken up room in read-write state zone bit, described level 2 buffering district, wherein, described read-write state zone bit is for identifying the current state in described level 2 buffering district; The timestamp of described first data represents the timestamp of first data of preserving in described level 2 buffering district; The timestamp of described latest data represents the timestamp of latest data; The size of having stored data content in the described described level 2 buffering district that taken up room large small records; The packet in described level 2 buffering district is containing timestamp and the data value of map file point.
4. method as claimed in claim 1 or 2, is characterized in that, described step 3 specifically comprises following sub-step:
(3-1) according to the state flag bit of the interim archive file of today, judge that whether described interim archive file is in idle condition, if it is perform step (3-2), otherwise execution step (3-3);
(3-2) state flag bit of described interim archive file is updated to write state;
(3-3) obtain an archives point numbering, and from described interim index file, read the interim indexing units of described archives point according to described archives point numbering, utilize the current data block first address in described interim indexing units, from described interim archive file, read the data field size of described current data block, and according to described archives point numbering, obtain the address information of its L2 cache;
(3-4) according to the size that takes up room, the data field size of current data block and the upper limit of data block in the buffer zone head in the level 2 buffering district in read states corresponding to described archives point, can judgement current data block hold all data in the described level 2 buffering district in read states, if can, perform step (3-6), otherwise execution step (3-5);
(3-5) afterbody at described interim archive file newly creates a data block, offset address by the new data block creating of the Next pointed of described current data block in interim archive file, and the current data block first address in the described interim indexing units in described interim index file is updated to the offset address of described new establishment data block in described interim archive file by described archives point;
(3-6) data in the described level 2 buffering district in read states are write to current data block, by the described size that taken up room in the buffer zone head in the described level 2 buffering district in read states, upgrade the data field size of current data block, the timestamp of described first data in the described level 2 buffering district in read states,, the timestamp of described latest data and the described size that taken up room are all initialized as 0, is continued to execution step (3-3) until all archives points are all finished dealing with.
5. method as claimed in claim 4, it is characterized in that, described interim archive file include file Tou He filing data district, described file header comprises described state flag bit and file size, the described interim archive file of described state flag bit explanation is current in what state, and described file size comprises the size of described filing data district and described file header.
6. method as claimed in claim 5, it is characterized in that, the described filing data district of described interim archive file is comprised of a plurality of archives point data blocks, Historical archiving data in the corresponding archives point a period of time section of each archives point data block, each archives point can comprise a plurality of archives point data blocks, described archives point data block comprises data field size, Next pointer and data field, wherein, the data of described data area stores archives point, described data field size is the shared storage size of data in described data field, described Next pointer is next archives point data block offset address in described interim archive file, thereby form an archives point data block chained list.
7. method as claimed in claim 4, it is characterized in that, it in described interim index file, is an interim indexing units of each archives point distribution, in described interim indexing units, store archives and putting first address and the current data block first address of first data block, wherein, the first address of first data block refers to the offset address of described first data block in described interim archive file, current data block first address refer to current just the data block of filing data can be for the region of data writing the offset address in described interim archive file.
8. method as claimed in claim 4, is characterized in that, described step 4 specifically comprises following sub-step:
(4-1) whether system Real-Time Monitoring current time is to interim archive file data preparation is write to the time threshold T of formal archive file
orderif arrive described time threshold T
orderperform step (4-2), otherwise continue Real-Time Monitoring current time;
(4-2) the packed data capacity sum of all files point in the interim archive file of scanning yesterday, according to formal archive file maxsize, judge whether current formal archive file can hold, to perform step (4-5), otherwise execution step (4-3);
(4-3) according to archives point numbering order from small to large, arrange the data in the interim archive file of yesterday, the data of each archives point corresponding data piece in described interim archive file are integrated together according to time sequencing, obtain the archives point compressed data set of a day;
(4-4) according to archives point numbering order from small to large, the compressed data set of each archives point is written to current formal archive file one by one, simultaneously, in corresponding filing index file, increase a filing index data piece, offset address and the length thereof of the compressed data set that records each archives point in current formal archive file, then, at date index file afterbody, increase a date index data item, offset address and the corresponding date thereof of the filing index data piece that record newly increases in current filing index file, wherein, described date index file consists of a plurality of index blocks, the corresponding formal archive file of each index block,
(4-5) a newly-built empty formal archive file and corresponding filing index file thereof, and insert corresponding index block at the afterbody of described date index file, using the formal archive file newly creating as current formal archive file, data set is write to newly-built archive file and upgrade described date index file and described filing index file;
(4-6) when the interim archive file of yesterday arrange complete after, the interim archive file of yesterday and interim index file can be deleted, and the interim archive file of newly-built tomorrow and interim index file are to guarantee that every day, temporary file was accomplished seamless switching in across day data storage.
9. the historical data indexing means based on historical data storage method described in claim 1, is characterized in that, comprising:
Step 1 is obtained an inquiry request, the initial time T that it comprises archives point numbering, inquiry
bEGINwith end time T
eND;
Step 2 is divided into N the subquery task in same nature sky by the query time section comprising in described inquiry request by nature sky, and wherein N represents the nature day number that described query time section comprises;
Step 3 is obtained a unenforced subquery task;
Step 4, according to " year-month-day " information of the initial time of described subquery task, judges whether initial time is today, if it is performs step 5, otherwise execution step 7;
Step 5 is calculated the offset address of interim indexing units correspondence in the interim index file of today of archives point according to the archives point numbering in described subquery task, computing formula is as follows:
Interim indexing units offset address=sizeof (interim indexing units) * (archives point numbering-1)
Wherein, sizeof (interim indexing units) represents the length of the interim indexing units of each archives point; According to the offset address of described interim indexing units, from the interim index file of today, read the interim indexing units of described archives point, thereby obtain first address and the current data block first address of first data block;
Step 6 is from the first data block of described archives point, travel through whole data block chained list, data field in read block be cached to compressed data set Data in order, also read the packed data in the current level 2 buffering district in writing state that described archives point is corresponding and be cached to compressed data set Data, continuing execution step 9;
The index block of " year-month-day " acquisition of information date index file of the initial time that step 7 comprises according to described subquery task, further from described index block, obtain formal archive file stores position and filename, corresponding filing index file storage directory and filename, and date index data item corresponding to Query Dates, further according to the filing index data in described date index data item, offset address and the archives point in correspondence filing index file numbered, obtain the index data of this day described archives point of Query Dates, offset address and the length of the data that comprise described Query Dates this day in the formal archive file of correspondence,
The index information that step 8 obtains according to described step 7 reads out described archives and puts this day corresponding compressed data set Data from corresponding formal archive file;
Step 9, according to the Data Update cycle of described archives point, decompresses to the data in described compressed data set Data, obtains in chronological sequence tactic described archives and puts daylong decompressed data set DeData;
If the initial time of the described subquery task of step 10 equals the end time, represent the data of a special time stamp of inquiry, in described decompressed data collection DeData, search the data D that timestamp equals initial time
aif there is return data D
aotherwise return to empty result set, if described initial time is not equal to the described end time, the data that represent a special time period of inquiry, by in described decompressed data collection DeData, meet data that timestamp is more than or equal to described initial time and is less than or equal to described end time condition in chronological sequence order be inserted into the set of subquery task result, by i sub-query task T
iinquire about resulting result data collection and be designated as RS
i, wherein, the span of i is 1～N;
Step 11 judges whether that the equal executed of all subquery tasks is complete, if not continuing to carry out described step 3, otherwise execution step 12;
Step 12 is by the resulting result data collection of all subquery tasks RS
iaccording to time order and function order, merge, obtain archives point to be checked at described initial time T
bEGINto described end time T
eNDall historical datas in time period, finally return to the net result collection after merging.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410364895.7A CN104090987B (en) | 2014-07-28 | 2014-07-28 | A kind of history data store and indexing means |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410364895.7A CN104090987B (en) | 2014-07-28 | 2014-07-28 | A kind of history data store and indexing means |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104090987A true CN104090987A (en) | 2014-10-08 |
| CN104090987B CN104090987B (en) | 2018-03-13 |
Family
ID=51638703
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410364895.7A Active CN104090987B (en) | 2014-07-28 | 2014-07-28 | A kind of history data store and indexing means |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104090987B (en) |
Cited By (38)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104408041A (en) * | 2014-10-13 | 2015-03-11 | 深圳市德传技术有限公司 | A method for storing GPS data |
| CN104679821A (en) * | 2014-12-30 | 2015-06-03 | 深圳市兰丁科技有限公司 | Data access processing method and data access processing device |
| CN104699816A (en) * | 2015-03-25 | 2015-06-10 | 成都华迈通信技术有限公司 | Method and device for processing data |
| CN104881481A (en) * | 2015-06-03 | 2015-09-02 | 安科智慧城市技术（中国）有限公司 | Method and device for accessing mass time sequence data |
| CN105426128A (en) * | 2015-11-16 | 2016-03-23 | 浪潮集团有限公司 | Index maintenance method and device |
| WO2016101659A1 (en) * | 2014-12-22 | 2016-06-30 | 华为技术有限公司 | Service data query method and apparatus and database system |
| CN105739912A (en) * | 2014-12-11 | 2016-07-06 | 北京北方微电子基地设备工艺研究中心有限责任公司 | Method and system for storing real-time data in semiconductor device |
| CN105792163A (en) * | 2014-12-24 | 2016-07-20 | 联芯科技有限公司 | Updating method and updating device of sub-band search association table, cell search method and cell search device |
| CN105786975A (en) * | 2016-02-04 | 2016-07-20 | 北京中盛益华科技有限公司 | Massive video storage method used for quick search |
| CN106021369A (en) * | 2016-05-11 | 2016-10-12 | 上海佳依佳信息科技有限公司 | Method and equipment for data change management |
| CN106095828A (en) * | 2016-06-01 | 2016-11-09 | 积成电子股份有限公司 | A kind of curve data storage scheme based on power information acquisition terminal |
| CN106557272A (en) * | 2015-09-30 | 2017-04-05 | 中国科学院软件研究所 | A kind of efficient sensor historic data archiving method |
| CN106815227A (en) * | 2015-11-27 | 2017-06-09 | 阿里巴巴集团控股有限公司 | A kind of method and apparatus of storage information |
| CN106855890A (en) * | 2017-01-09 | 2017-06-16 | 广州巨杉软件开发有限公司 | A kind of method for realizing the final consistency full-text search of high-performance data storehouse |
| CN107250996A (en) * | 2015-03-24 | 2017-10-13 | 英特尔公司 | compacting for memory hierarchy |
| CN107273403A (en) * | 2017-04-24 | 2017-10-20 | 西安电子科技大学 | A kind of history data collection and the method for storage |
| CN107562764A (en) * | 2016-07-01 | 2018-01-09 | 中国铁路信息技术中心 | A kind of car flow of railway data slicer method and apparatus |
| CN107894875A (en) * | 2017-11-20 | 2018-04-10 | 深圳市德传技术有限公司 | A kind of RTU date storage methods |
| CN108008431A (en) * | 2017-11-28 | 2018-05-08 | 佛山市安尔康姆航空科技有限公司 | A kind of unmanned vehicle GPS location recording method |
| CN108614818A (en) * | 2016-12-09 | 2018-10-02 | 杭州海康威视数字技术股份有限公司 | A kind of storage of data, update and querying method and device |
| CN108920553A (en) * | 2018-06-19 | 2018-11-30 | 西安电子科技大学 | A kind of data record method for onboard multi-sensor platform |
| CN109118609A (en) * | 2017-06-26 | 2019-01-01 | 深圳市道通智能航空技术有限公司 | Unmanned plane during flying log recording method, device, storage medium and computer equipment |
| CN110704174A (en) * | 2018-07-09 | 2020-01-17 | 中国移动通信有限公司研究院 | Memory release method and device, electronic equipment and storage medium |
| CN110765074A (en) * | 2019-09-20 | 2020-02-07 | 国网山东省电力公司青岛供电公司 | Method and system for quickly accessing electric load curve data of acquisition terminal |
| CN111061736A (en) * | 2019-12-16 | 2020-04-24 | 北京盛世华人供应链管理有限公司 | Multi-index cache big data overtemperature judgment algorithm for logistics temperature control industry |
| CN111352935A (en) * | 2019-07-29 | 2020-06-30 | 阿里巴巴集团控股有限公司 | Index creating method, device and equipment in block chain type account book |
| CN111881138A (en) * | 2020-07-29 | 2020-11-03 | 北京微步在线科技有限公司 | Data storage method and device |
| CN112214493A (en) * | 2020-10-20 | 2021-01-12 | 北京九思易自动化软件有限公司 | Historical record storage method and device, electronic equipment and storage medium |
| CN112639746A (en) * | 2018-09-07 | 2021-04-09 | 株式会社东芝 | Database device, program, and data processing method |
| CN112685329A (en) * | 2019-10-17 | 2021-04-20 | 云丁网络技术(北京)有限公司 | Method for processing data and related device |
| CN112702418A (en) * | 2020-12-21 | 2021-04-23 | 潍柴动力股份有限公司 | Double-cache data downloading control method and device and vehicle |
| CN114116795A (en) * | 2021-11-01 | 2022-03-01 | 北京力控元通科技有限公司 | Data storage and query method, device, storage medium and electronic equipment |
| CN115658619A (en) * | 2022-11-17 | 2023-01-31 | 北京星辰天合科技股份有限公司 | Process processing method and device, processor and electronic equipment |
| CN116204684A (en) * | 2023-02-01 | 2023-06-02 | 浙江正泰仪器仪表有限责任公司 | Storage method, device, equipment and medium of electric energy meter |
| CN116301579A (en) * | 2022-12-06 | 2023-06-23 | 无锡市海鹰加科海洋技术有限责任公司 | File access system for self-contained measuring equipment and access method thereof |
| CN116340367A (en) * | 2023-05-30 | 2023-06-27 | 杭州芯翼科技有限公司 | Data query method, device, equipment and storage medium |
| CN116383165A (en) * | 2023-06-03 | 2023-07-04 | 浙江档安科技有限公司 | Adaptive writing archive storage method |
| CN116541348A (en) * | 2023-03-22 | 2023-08-04 | 河北热点科技股份有限公司 | Intelligent data storage method and terminal query integrated machine |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110837592B (en) * | 2019-11-08 | 2021-01-26 | 京东数字科技控股有限公司 | Method, apparatus and computer readable storage medium for data archiving |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030149702A1 (en) * | 2002-02-05 | 2003-08-07 | Compaq Information Technologies Group, L.P. | Operational data store |
| CN102646130A (en) * | 2012-03-12 | 2012-08-22 | 华中科技大学 | Method for storing and indexing mass historical data |
| CN103853499A (en) * | 2012-11-30 | 2014-06-11 | 上海可鲁系统软件有限公司 | Multi-source data storing and reading method |
-
2014
- 2014-07-28 CN CN201410364895.7A patent/CN104090987B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030149702A1 (en) * | 2002-02-05 | 2003-08-07 | Compaq Information Technologies Group, L.P. | Operational data store |
| CN102646130A (en) * | 2012-03-12 | 2012-08-22 | 华中科技大学 | Method for storing and indexing mass historical data |
| CN103853499A (en) * | 2012-11-30 | 2014-06-11 | 上海可鲁系统软件有限公司 | Multi-source data storing and reading method |
Non-Patent Citations (1)
| Title |
|---|
| 王章: "海量历史数据库系统研究", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
Cited By (59)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104408041A (en) * | 2014-10-13 | 2015-03-11 | 深圳市德传技术有限公司 | A method for storing GPS data |
| CN104408041B (en) * | 2014-10-13 | 2018-03-30 | 深圳市德传技术有限公司 | A kind of store method of gps data |
| CN105739912A (en) * | 2014-12-11 | 2016-07-06 | 北京北方微电子基地设备工艺研究中心有限责任公司 | Method and system for storing real-time data in semiconductor device |
| WO2016101659A1 (en) * | 2014-12-22 | 2016-06-30 | 华为技术有限公司 | Service data query method and apparatus and database system |
| CN105792163A (en) * | 2014-12-24 | 2016-07-20 | 联芯科技有限公司 | Updating method and updating device of sub-band search association table, cell search method and cell search device |
| CN104679821A (en) * | 2014-12-30 | 2015-06-03 | 深圳市兰丁科技有限公司 | Data access processing method and data access processing device |
| CN104679821B (en) * | 2014-12-30 | 2018-12-21 | 深圳市海蕴新能源有限公司 | data access processing method and device |
| CN107250996A (en) * | 2015-03-24 | 2017-10-13 | 英特尔公司 | compacting for memory hierarchy |
| CN107250996B (en) * | 2015-03-24 | 2022-01-11 | 英特尔公司 | Method and apparatus for compaction of memory hierarchies |
| CN104699816B (en) * | 2015-03-25 | 2017-12-12 | 成都华迈通信技术有限公司 | A kind of data processing method and device |
| CN104699816A (en) * | 2015-03-25 | 2015-06-10 | 成都华迈通信技术有限公司 | Method and device for processing data |
| CN104881481A (en) * | 2015-06-03 | 2015-09-02 | 安科智慧城市技术（中国）有限公司 | Method and device for accessing mass time sequence data |
| CN104881481B (en) * | 2015-06-03 | 2018-05-01 | 广东省机电设备招标有限公司 | A kind of method and device for accessing magnanimity time series data |
| CN106557272A (en) * | 2015-09-30 | 2017-04-05 | 中国科学院软件研究所 | A kind of efficient sensor historic data archiving method |
| CN105426128A (en) * | 2015-11-16 | 2016-03-23 | 浪潮集团有限公司 | Index maintenance method and device |
| CN106815227A (en) * | 2015-11-27 | 2017-06-09 | 阿里巴巴集团控股有限公司 | A kind of method and apparatus of storage information |
| CN105786975A (en) * | 2016-02-04 | 2016-07-20 | 北京中盛益华科技有限公司 | Massive video storage method used for quick search |
| CN105786975B (en) * | 2016-02-04 | 2018-12-28 | 北京中盛益华科技有限公司 | Massive video storage method for quick-searching |
| CN106021369A (en) * | 2016-05-11 | 2016-10-12 | 上海佳依佳信息科技有限公司 | Method and equipment for data change management |
| CN106095828A (en) * | 2016-06-01 | 2016-11-09 | 积成电子股份有限公司 | A kind of curve data storage scheme based on power information acquisition terminal |
| CN106095828B (en) * | 2016-06-01 | 2019-03-29 | 积成电子股份有限公司 | A kind of curve data storage scheme based on power information acquisition terminal |
| CN107562764A (en) * | 2016-07-01 | 2018-01-09 | 中国铁路信息技术中心 | A kind of car flow of railway data slicer method and apparatus |
| CN108614818A (en) * | 2016-12-09 | 2018-10-02 | 杭州海康威视数字技术股份有限公司 | A kind of storage of data, update and querying method and device |
| CN108614818B (en) * | 2016-12-09 | 2020-12-25 | 杭州海康威视数字技术股份有限公司 | Data storage, updating and query method and device |
| CN106855890A (en) * | 2017-01-09 | 2017-06-16 | 广州巨杉软件开发有限公司 | A kind of method for realizing the final consistency full-text search of high-performance data storehouse |
| CN106855890B (en) * | 2017-01-09 | 2020-07-28 | 深圳巨杉数据库软件有限公司 | Method for realizing final consistency full-text retrieval of high-performance database |
| CN107273403A (en) * | 2017-04-24 | 2017-10-20 | 西安电子科技大学 | A kind of history data collection and the method for storage |
| CN109118609A (en) * | 2017-06-26 | 2019-01-01 | 深圳市道通智能航空技术有限公司 | Unmanned plane during flying log recording method, device, storage medium and computer equipment |
| CN109118609B (en) * | 2017-06-26 | 2020-11-13 | 深圳市道通智能航空技术有限公司 | Unmanned aerial vehicle flight log recording method and device, storage medium and computer equipment |
| CN107894875A (en) * | 2017-11-20 | 2018-04-10 | 深圳市德传技术有限公司 | A kind of RTU date storage methods |
| CN107894875B (en) * | 2017-11-20 | 2020-12-04 | 深圳市德传技术有限公司 | RTU data storage method |
| CN108008431A (en) * | 2017-11-28 | 2018-05-08 | 佛山市安尔康姆航空科技有限公司 | A kind of unmanned vehicle GPS location recording method |
| CN108920553A (en) * | 2018-06-19 | 2018-11-30 | 西安电子科技大学 | A kind of data record method for onboard multi-sensor platform |
| CN108920553B (en) * | 2018-06-19 | 2020-10-30 | 西安电子科技大学 | Data recording method for airborne multi-sensor platform |
| CN110704174A (en) * | 2018-07-09 | 2020-01-17 | 中国移动通信有限公司研究院 | Memory release method and device, electronic equipment and storage medium |
| CN112639746A (en) * | 2018-09-07 | 2021-04-09 | 株式会社东芝 | Database device, program, and data processing method |
| CN111352935A (en) * | 2019-07-29 | 2020-06-30 | 阿里巴巴集团控股有限公司 | Index creating method, device and equipment in block chain type account book |
| CN110765074A (en) * | 2019-09-20 | 2020-02-07 | 国网山东省电力公司青岛供电公司 | Method and system for quickly accessing electric load curve data of acquisition terminal |
| CN110765074B (en) * | 2019-09-20 | 2022-03-01 | 国网山东省电力公司青岛供电公司 | Method and system for quickly accessing electric load curve data of acquisition terminal |
| CN112685329A (en) * | 2019-10-17 | 2021-04-20 | 云丁网络技术(北京)有限公司 | Method for processing data and related device |
| CN111061736A (en) * | 2019-12-16 | 2020-04-24 | 北京盛世华人供应链管理有限公司 | Multi-index cache big data overtemperature judgment algorithm for logistics temperature control industry |
| CN111061736B (en) * | 2019-12-16 | 2023-05-09 | 北京盛世华人供应链管理有限公司 | Multi-index cache big data overtemperature judging method in logistics temperature control industry |
| CN111881138A (en) * | 2020-07-29 | 2020-11-03 | 北京微步在线科技有限公司 | Data storage method and device |
| CN112214493A (en) * | 2020-10-20 | 2021-01-12 | 北京九思易自动化软件有限公司 | Historical record storage method and device, electronic equipment and storage medium |
| CN112702418A (en) * | 2020-12-21 | 2021-04-23 | 潍柴动力股份有限公司 | Double-cache data downloading control method and device and vehicle |
| CN114116795A (en) * | 2021-11-01 | 2022-03-01 | 北京力控元通科技有限公司 | Data storage and query method, device, storage medium and electronic equipment |
| CN114116795B (en) * | 2021-11-01 | 2022-12-20 | 北京力控元通科技有限公司 | Data storage and query method, device, storage medium and electronic equipment |
| CN115658619B (en) * | 2022-11-17 | 2023-03-28 | 北京星辰天合科技股份有限公司 | Process processing method and device, processor and electronic equipment |
| CN115658619A (en) * | 2022-11-17 | 2023-01-31 | 北京星辰天合科技股份有限公司 | Process processing method and device, processor and electronic equipment |
| CN116301579A (en) * | 2022-12-06 | 2023-06-23 | 无锡市海鹰加科海洋技术有限责任公司 | File access system for self-contained measuring equipment and access method thereof |
| CN116301579B (en) * | 2022-12-06 | 2024-01-12 | 无锡市海鹰加科海洋技术有限责任公司 | File access system for self-contained measuring equipment and access method thereof |
| CN116204684A (en) * | 2023-02-01 | 2023-06-02 | 浙江正泰仪器仪表有限责任公司 | Storage method, device, equipment and medium of electric energy meter |
| CN116204684B (en) * | 2023-02-01 | 2024-06-04 | 浙江正泰仪器仪表有限责任公司 | Storage method, device, equipment and medium of electric energy meter |
| CN116541348A (en) * | 2023-03-22 | 2023-08-04 | 河北热点科技股份有限公司 | Intelligent data storage method and terminal query integrated machine |
| CN116541348B (en) * | 2023-03-22 | 2023-09-26 | 河北热点科技股份有限公司 | Intelligent data storage method and terminal query integrated machine |
| CN116340367A (en) * | 2023-05-30 | 2023-06-27 | 杭州芯翼科技有限公司 | Data query method, device, equipment and storage medium |
| CN116340367B (en) * | 2023-05-30 | 2023-08-08 | 杭州芯翼科技有限公司 | Data query method, device, equipment and storage medium |
| CN116383165A (en) * | 2023-06-03 | 2023-07-04 | 浙江档安科技有限公司 | Adaptive writing archive storage method |
| CN116383165B (en) * | 2023-06-03 | 2023-08-11 | 浙江档安科技有限公司 | Adaptive writing archive storage method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104090987B (en) | 2018-03-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104090987A (en) | Historical data storage and indexing method | |
| CN102890722B (en) | Indexing method applied to time sequence historical database | |
| CN103605805B (en) | A kind of storage method of magnanimity time series data | |
| CN102646130B (en) | Method for storing and indexing mass historical data | |
| CN107423422B (en) | Spatial data distributed storage and search method and system based on grid | |
| CN101777016B (en) | Snapshot storage and data recovery method of continuous data protection system | |
| CN103366015B (en) | A kind of OLAP data based on Hadoop stores and querying method | |
| CN103412803B (en) | The method and device that data are recovered | |
| US9047330B2 (en) | Index compression in databases | |
| CN103488684B (en) | Electric reliability index quick calculation method based on data cached multiple threads | |
| CN103488709A (en) | Method and system for building indexes and method and system for retrieving indexes | |
| CN107463447B (en) | B + tree management method based on remote direct nonvolatile memory access | |
| CN101667205B (en) | Method for memorizing real time measure point data for quick review | |
| CN107870970B (en) | A kind of data store query method and system | |
| CN103176754A (en) | Reading and storing method for massive amounts of small files | |
| CN105320775A (en) | Data access method and apparatus | |
| CN104866497A (en) | Metadata updating method and device based on column storage of distributed file system as well as host | |
| CN102129435A (en) | Data storage service control method and system | |
| CN103186617B (en) | A kind of method and apparatus storing data | |
| CN108491402B (en) | Method for implementing lightweight file system | |
| US11741073B2 (en) | Granularly timestamped concurrency control for key-value store | |
| CN103853499A (en) | Multi-source data storing and reading method | |
| Lu et al. | TridentKV: A read-optimized LSM-tree based KV store via adaptive indexing and space-efficient partitioning | |
| CN107273403A (en) | A kind of history data collection and the method for storage | |
| JPS63116232A (en) | Method of memorizing and retrieving data using computer |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |