CN111158604A - Internet of things time sequence data storage and retrieval method for flash memory particle array - Google Patents
Internet of things time sequence data storage and retrieval method for flash memory particle array Download PDFInfo
- Publication number
- CN111158604A CN111158604A CN201911404306.2A CN201911404306A CN111158604A CN 111158604 A CN111158604 A CN 111158604A CN 201911404306 A CN201911404306 A CN 201911404306A CN 111158604 A CN111158604 A CN 111158604A
- Authority
- CN
- China
- Prior art keywords
- data
- block
- flash memory
- logical page
- time
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
- G06F3/0679—Non-volatile semiconductor memory device, e.g. flash memory, one time programmable memory [OTP]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0629—Configuration or reconfiguration of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/0652—Erasing, e.g. deleting, data cleaning, moving of data to a wastebasket
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
- G06F3/0656—Data buffering arrangements
Abstract
The invention discloses a storage and retrieval method of internet of things time sequence data of a flash memory particle array, which is specially improved correspondingly for reading and writing of multi-type time sequence data. According to the life cycle of various data, establishing respective block-level address mapping tables for physical blocks of which the space is allocated in advance from a flash memory particle array, combining data into a storage unit by taking a page as a unit in a buffer area of a memory, caching a certain amount of data, then writing the data into the physical blocks allocated to the memory in advance, writing a page of data from a first logic block to the inside in sequence during writing, and ensuring the same offset address of each block after one round of writing data; and establishing a sensor change condition recording table to record parameters such as data time points, logical page numbers, logical block numbers and the like, and calculating the address of the target data according to the parameters brought into the sensor change condition recording table by the time points of the target data during data query to obtain data.
Description
Technical Field
The invention relates to the field of storage and retrieval of databases, in particular to a method for storing and retrieving time series data of an internet of things of a flash memory particle array.
Background
With the continuous popularization of the internet of things technology and the continuous development of the wireless sensor network technology, more and more sensors are used in the layout of the sensor network, the sampling rate of the sensors is higher and higher, the generated data volume is larger and larger, the main characteristics of the data are time continuity and various types of data, and how to rapidly store and retrieve the various types of data becomes a problem to be solved. Especially, for the rapid retrieval of specified data, the timeliness of data retrieval under large data scale cannot be completely met by a common relational database; and the time relation in the time sequence data can not be well utilized by the novel non-relational database, the retrieval efficiency directly depends on hardware investment, and the cost performance is not high.
Because the time series data of the internet of things are generated by the acquisition of the sensors, a large number of sensors are required for the detection of one region to acquire data, and the data are generated periodically and continuously and have the characteristic of massive data. For the storage of mass data, the traditional database is limited by the read-write speed of a disk on one hand, and on the other hand, a complex index needs to be constructed for retrieving data, the index is inconvenient to update and maintain, and a bottleneck exists for the storage and retrieval of mass data.
The flash memory particle array has more read-write speed than the magnetic disk block due to the parallel advantages of the structure of the flash memory particle array and the time waste of address seeking and the like when the magnetic disk rotates during reading and writing data, and is suitable for a scene of fast reading and writing application of a large amount of data. The flash memory generally adopts a strategy of updating in different places, namely writing data to be updated in physical pages at other places, reconstructing mapping relation, and marking the original data pages as invalid for erasing operation in the future. For data deletion, flash memory is operated in units of blocks, and it is necessary to transfer valuable data in a block to a new physical page and then erase the physical block. Each block is limited by the erasing times, the block becomes a bad block and cannot be used after the erasing reaches a certain number, when a block is judged to be a bad block through an ECC processing mechanism, the block is marked as being unusable so as to be isolated, and the bad blocks are not selected when a corresponding mapping relation is established.
A general solid state disk is a typical flash memory array, and in order to maintain the general versatility of the conventional solid state disk, i.e. to prolong the service life of the flash memory, a complex garbage collection and wear leveling mechanism is required to reduce the erasing operation of the flash memory and increase the service life of the flash memory. The use efficiency of the flash memory is influenced to a certain extent by the complex mechanisms, and the mechanisms are optimized correspondingly according to the characteristics and the distribution mode of the time sequence data in the flash memory array. The existing method only considers the same data to be placed in the same physical block, combines the physical blocks into a plurality of data files to store the data, does not carry out corresponding optimized storage on the physical address of the data stored on a storage medium, and has poor random reading and writing efficiency if a plurality of physical blocks in the file are too dispersed on a magnetic disk and poor reading and writing efficiency if a plurality of physical blocks storing continuous data are concentrated in one flash memory particle for a flash memory array.
In addition, the existing method reserves a data storage space to prevent the increase of acquisition points, the condition of the increase of the acquisition data lacks effective utilization of the storage space, if the number of the acquisition points of the specified type is not increased, the overflow data area cannot be used for writing other types of data, and the utilization efficiency of the storage space is lower. For data retrieval, the existing method searches data files corresponding to a time period first, and then compares the time difference of each data in the data files to find data corresponding to a time point.
Disclosure of Invention
Aiming at the defects in the prior art, the storage and indexing method of the Internet of things time sequence data of the flash memory particle array provided by the invention is correspondingly improved for the reading and writing of the multi-type time sequence data.
In order to achieve the above purpose, the invention adopts the technical scheme that:
the scheme provides a method for storing and retrieving time series data of the Internet of things of a flash memory particle array, which comprises the following steps:
s1, according to the life cycle and unit time sampling rate of the stored data and the size of single time point data, allocating physical blocks for the flash memory particles in the flash memory particle array in advance from different flash memory particles, and constructing a corresponding block level address mapping table;
s2, according to the block-level address mapping table, constructing a storage unit with the size of a page in a buffer area of a memory, and writing data into the storage unit in a time sequence;
s3, creating and updating a sensor change condition recording table according to the real-time monitoring of the number and sampling rate of the sensors and the block-level address mapping table, and recording the relation parameters of the data time point and the logical page number and the relation parameters between the logical page number and the logical block number;
s4, inquiring the last relational parameter in the sensor change condition record table, substituting the relational parameter into an offset address obtained by calculation and writing data into a flash memory, and finishing the storage of the time series data of the Internet of things;
and S5, calculating to obtain the address of the target time point data according to the relation parameter of the position of the starting time closest to the target time point in the query sensor change condition record table, and completing retrieval of the time series data of the Internet of things.
Further, in the step S1, the flash memory particle array includes the type of data collected by the sensor, the start time t0Start time logical page lpnfirstNumber of added blocks i, logical page lpniLogic block lbniNumber of blocks before addition Xi-1Current total number of blocks XiNumber of sensors m and samplingThe sample rate p.
Still further, in step S1, each physical block is allocated in a different flash memory, and each physical block only stores one type of data, and an independent block-level address mapping table is constructed for each type of data.
Still further, the sensor change condition record table in step S3 includes the number and the sampling rate of the current type data sensors, the start time at the current number and the sampling rate, and the logical page number where the start time is located.
Still further, the updating conditions of the sensor change condition record table in the step S3 are as follows:
updating the sensor change log table as the number of sensors increases or decreases, wherein:
when the number of sensors is reduced, the current total block number X is not updatedi；
When the number of sensors is increased and a new physical block needs to be added for saving data, the current total block number X is updatedi。
Still further, the step S4 includes the following steps:
s401, retrieving data of a target time point according to the relation parameters obtained in the step S3 and the sensor change condition record table;
s402, inquiring the initial time closest to the target time point, and calculating according to the closest initial time and the relation parameters of the record list time point and the logical page to obtain a target logical page number;
s403, comparing the target logical page with the latest logical page number recorded in the sensor change condition record table to calculate a difference value;
s404, calculating by using the difference to obtain a logic block number where the target time point data is located and offset thereof;
s405, obtaining an offset address in a physical block where the target data is located according to the logical block number where the target time point data is located and the offset thereof, and completing data storage of the Internet of things time sequence based on the flash memory particle array.
Still further, the expression of the target logical page number in step S402 is as follows:
wherein LPN represents the target logical page number, tnRepresents a target time point, t0Representing the last point in time before the change recorded by the sensor change table, m representing the number of sensors after the change, p representing the sampling rate of the sensors, lpnfirstIndicating the last logical page number of the data deposit.
Still further, in step S404, the expression of the logical block number LBN where the target point-in-time data is located is as follows:
lpnlasti＝lpni+Xi-lbni
wherein i represents the number of times of adding a new block to the record table, the initial value is 0, which represents the X block physical block allocated to the data just after the start, and the subsequent addition when the physical block needs to be additionally configured for the newly added sensor, lpn represents the logical page number of the target time point, lpnlasti represents the logical page number of the last offset bit written in the old block before adding the new block, lpniIndicating the logical page number to which the current data of the data was written when the block was added i, lbniIndicates the logical block number, X, in which it is locatediRepresenting the total number of blocks allocated to it after the ith addition of a block.
Still further, in step S404, the expression of the offset at the target time point data is as follows:
lpn denotes a logical page where the target time point is locatedNumber lpnlastiLogical page number, X, representing the last offset bit written in the old block before the addition of the new blockiDenotes the total number of blocks, X, allocated to the block after the ith additionnIndicates the total number of blocks after the addition of the block n, and k indicates the number of times the block is added.
The invention has the beneficial effects that:
(1) the invention establishes a plurality of mapping tables by classification, writes data into physical addresses of continuous logical address mapping in the tables sequentially, and preferentially uses physical blocks in different flash memory chips when the address mapping in each table is established, so that the parallelism characteristic of a flash memory array can be utilized to the maximum extent, the concurrent reading and writing speed of the data is effectively improved, linear relation parameters between time and logical pages in data query are recorded by a sensor change condition recording table, the position of the data can be quickly obtained by calculation, and the overhead of additionally constructing indexes for various data is avoided. Optimizing the data reading and writing speed in the storage based on the flash memory aiming at the time sequence data;
(2) for the storage of data, according to parameters such as a life cycle of the data, a unit time sampling rate, the size of a single time point and the like, allocating physical block addresses stored for the data in advance and constructing a corresponding block level address mapping table, in order to improve the reading and writing speed of the data, the physical address bit allocation written in the data is emphasized when the data is written, in order to achieve the purpose of quick reading and writing, according to the characteristic of a parallel structure of a flash memory array, the data is dispersedly written into the allocated physical blocks by taking pages as units, so that continuous data can be dispersedly stored in a plurality of physical blocks, the physical blocks are distributed in different flash memories, and therefore, when a large amount of continuous time sequence data is read, reading and writing can be simultaneously carried out from the plurality of flash memories, and the storage speed of the data is effectively accelerated;
(3) because the data has time correlation, the data characteristics show a sequential linear relation in time, the data is written into the corresponding logical address in time sequence, the position of the corresponding logical address can be quickly calculated according to the linear relation, and the target data stored on the flash memory physical address can be quickly found through the address mapping. Because the physical addresses are dispersed in each flash memory chip, reading a large amount of data in the range is equivalent to simultaneously reading data from a plurality of chips, and the parallelism of the flash memory array structure can be fully utilized to achieve the aim of quickly reading the data;
(4) for the condition that the number of sensors is increased, a new clean physical block needs to be added to a block-level address mapping table created before the data is stored, and the newly added physical block calls the clean physical block from flash memory particles which do not allocate the physical block to the data preferentially and adds the clean physical block to the address mapping table. By adopting the method, the overflow data storage space does not need to be reserved for each type of sensor data independently, and the data can be uniformly distributed as required when the storage blocks are required to be added, so that the utilization efficiency of the storage space is improved;
(5) the invention can ensure that the corresponding linear parameters are stored when the number of the sensors changes by constructing the respective sensor change condition recording tables, and the tables are added with new contents when the number of the sensors changes, thereby being convenient and fast to inquire which logical page the data of the time point to be inquired is positioned in, and avoiding independently constructing a page level table to inquire the address of the data. When the number of the sensors is increased to cause the need of additional storage of a new physical block, the sensors are distributed according to needs, and the sensor change condition recording table also records the use condition of the internal storage space of the physical block in which data is written before the new physical block is added, so that the sensor change condition recording table is used for quickly inquiring related parameters and obtaining the offset in the logical block corresponding to the logical page number through calculation;
(6) in order to ensure the normal writing and reading of data when the number of the sensors changes, a sensor change condition recording table is maintained, the table records the time when the number of the sensors changes and the written data scale of the sensors when the number of the sensors changes, the content of the table is divided into a time point and logic page relation part and a logic page and logic block relation part, and on the basis of the table, the storage position of the sensor data with the specified time and the specified type can be obtained by combining a block-level address mapping table through simple calculation;
(7) in the data acquisition of the Internet of things, as the data are regularly and uniformly deleted according to the stored life cycle, the invention abandons the complex garbage recovery mechanism in the general SSD and can meet the requirement only by maintaining a simple bad block table. Compared with the general SSD, the method also avoids the inter-block migration operation of the data, thereby avoiding the consumption of the service life of the flash memory caused by the data migration and indirectly prolonging the service life of the flash memory. For the retrieval operation of data, the storage position of the designated data can be quickly calculated according to the data writing time, the sensor change condition record table and the block-level address mapping table, so that the complicated address retrieval operation of the traditional database is avoided.
Drawings
FIG. 1 is a flow chart of the method of the present invention.
FIG. 2 is a diagram illustrating data distribution of a buffer and read/write distribution of a flash memory granule array according to the present invention.
Fig. 3 is a schematic diagram illustrating distribution of data in a block in the present embodiment.
FIG. 4 is a flow chart of adding new memory blocks and updating the sensor change status log table according to the requirement of the present invention.
Detailed Description
The following description of the embodiments of the present invention is provided to facilitate the understanding of the present invention by those skilled in the art, but it should be understood that the present invention is not limited to the scope of the embodiments, and it will be apparent to those skilled in the art that various changes may be made without departing from the spirit and scope of the invention as defined and defined in the appended claims, and all matters produced by the invention using the inventive concept are protected.
Examples
The invention discloses a storage and retrieval method of internet of things time sequence data of a flash memory particle array, which is specially improved correspondingly for reading and writing of multi-type time sequence data. According to the life cycle of various data, establishing respective block-level address mapping tables for physical blocks of which the space is allocated in advance from a flash memory particle array, combining data into a storage unit by taking a page as a unit in a buffer area of a memory, caching a certain amount of data, then writing the data into the physical blocks allocated to the memory in advance, writing a page of data from a first logic block to the inside in sequence during writing, and ensuring the same offset address of each block after one round of writing data; and establishing a sensor change condition recording table to record data time points and logical page numbers, and relation parameters between the logical page numbers and the logical block numbers, and quickly calculating the address of the target data according to all parameters brought into the sensor change condition recording table by the time points of the target data during data query to obtain data. As shown in fig. 1, the method comprises the following steps:
s1, according to the life cycle and unit time sampling rate of the stored data and the size of single time point data, allocating physical blocks for the flash memory particles in the flash memory particle array in advance from different flash memory particles, and constructing a corresponding block level address mapping table; the stored data comprises the data type collected by the sensor and the starting time t0Start time logical page lpnfirstNumber of added blocks i, logical page lpniLogic block lbniCurrent total number of blocks XiThe number m of sensors and the sampling rate p; each physical block is distributed in different flash memories, each physical block only stores one type of data, and an independent block-level address mapping table is constructed for each type of data;
s2, according to the block-level address mapping table, constructing a storage unit with the size of a page in a buffer area of a memory, and writing data into the storage unit in a time sequence;
and S3, creating and updating a sensor change condition recording table according to the real-time monitoring of the number and the sampling rate of the sensors and the block-level address mapping table, and recording the relation parameters of the data time point and the logical page number and the relation parameters between the logical page number and the logical block number.
In this embodiment, the data comes from different types of sensors in the sensor network, and the information included in each piece of data is the data type of the sensor, the sampling time of the data, and the data acquired by the sensor. The method comprises the steps of writing collected various data into a buffer area in a memory, storing the data in a time sequence by taking the size of a page as a unit, combining the data into a plurality of pages in the buffer area, then writing the pages into flash memory chips in a flash memory particle array, distributing stored physical blocks for the stored data in advance through the life cycle of data storage, distributing each block in different flash memories and storing only one type of data, constructing an independent block-level mapping table for each type of data, and writing the data into each physical block by taking the page as a unit. The continuous logic pages are distributed in each logic block, so that written data can be ensured to be written into the flash memory particle array dispersedly, the parallel structure of the flash memory particle array can be well utilized, the data can be written into the flash memory particle array in a concurrent mode, the bandwidth of a data bus can be fully utilized, and the data can be stored rapidly.
In this embodiment, as shown in fig. 2, fig. 2 is a data distribution of a memory buffer and a read-write distribution diagram of a flash memory grain array, and it can be seen from the diagram that the buffer in the memory allocates a corresponding buffer area for each type of data therein, the area of each data in the buffer is set to a size according to the sampling rate, the number of sensors and the pre-stored time, the data in the last period of time is stored in the buffer of each type of data, when the data in the last time point is to be queried, the data can be directly read from the memory, because the flash memory performs a single write in page unit, the data needs to be combined into a write unit of page size in the buffer, otherwise, the data is placed in the buffer, and a corresponding physical block is allocated for the type of data in advance according to the life cycle of the data, assuming that N physical blocks are allocated for a certain type of data, each physical block is individually located in one flash memory chip of the flash memory array, the method comprises the steps of writing N pages of the same-type data into a flash memory particle array every time, namely writing one page of data into each flash memory, marking a physical block where each page of the written-in data is located as the same-type data, and writing the data into the next offset page of the physical block when the same-type data is written next time, so that each data basically keeps an independent physical block in a plurality of flash memory chips by writing the data, and the offset of continuous time sequence data of the N pages in the physical block is ensured to be consistent through one-round page writing, and the linear mapping relation between the logical page and the logical block is convenient to construct. The method comprises the steps of constructing a corresponding block-level mapping table, for address allocation of each type of data, independently creating and maintaining one mapping table by a controller according to a mark of a data type in a buffer area for each type of sensor, distributing a clean physical block on each of N flash memory chips to construct the mapping table before data is written into the block-level mapping table of each type of data, wherein the writing amount of each type of data is different due to the difference of the number of sensors and the sampling rate, and the number of the physical blocks occupied by the address mapping table of each type of data is also different. The physical blocks of the mapping are selected from the physical blocks in different flash memories, so that the data can be quickly written by utilizing the parallelism of the flash memory array to the maximum extent, because the data of the flash memories are deleted by taking the blocks as units, the mapping relation of other types of data cannot be influenced by erasing a certain physical block when the data are deleted, and the garbage recycling work of the flash memories can be facilitated.
In this embodiment, as shown in fig. 3, since the number of sensors is not fixed, and the amount of data collected may be changed due to a fault, etc., which may cause a change in the linear relationship between the time point of searching data and the logical page number, because the logical page number is calculated by the time point when searching data, and then the position of the page address where the data is located in the block is obtained by the address allocation method between the logical page and the logical block, when the number of sensors does not change, the corresponding logical page number can be calculated by a fixed linear relationship, when the number of sensors changes, the corresponding query relationship needs to be reconstructed, and at this time, a sensor change condition recording table needs to be separately established for each type of data, where the information included in the table includes the number of sensors of the current type, the sampling rate, the start time of the current number and the sampling rate, and the current logical page number, the table automatically generates a new line of data when the change of the number of the sensors is detected, because the change of the number of the sensors does not occur frequently, the table is convenient to maintain and update daily, time and logic pages are recorded in the table, and each parameter of a linear relation between the logic pages and the logic blocks is convenient for finding data corresponding to a time point quickly during later query. The continuous time data are stored in continuous logical pages, and the written data are dispersed in one flash memory chip of the flash memory particle array, so that the parallel structure of the flash memory particle array can be maximally utilized for writing and reading the data by the writing mode. The relation between the logical page and the logical block can be changed because the number of the sensors is increased and extra physical blocks are added for storing extra data, the traditional logical page and the logical block are statically allocated to determine the logical block in which the logical page is positioned, so that the number of blocks used for storing is required to be known in advance, the expansibility is not good, the relation parameters between the logical page and the logical block are recorded through a sensor change condition recording table, if N blocks exist in a mapping table before the number of the sensors is increased and new physical blocks are added behind the mapping table for storing data, the using condition of space of the previous N blocks is recorded through an internal storage table, the logical block number corresponding to the last logical page number before the new physical block is added and the offset in the logical block are recorded in the table, and the information is added into the table when the new physical block is added for storing data, and adding the new physical block to the data sequentially written into each physical block sequentially in sequence later by taking the address offset of the new data written into the last logical page before as a reference, and circulating by the aid of the offset, so that the calculation relationship between the logical pages and the blocks can be quickly constructed through the table.
In the embodiment, the corresponding linear parameters can be stored when the number of the sensors changes by constructing the respective sensor change condition recording tables, and new contents are added to the tables when the number of the sensors changes, so that the logical page where the data of the time point to be checked is located can be conveniently and quickly inquired, and the condition that one page level table is independently constructed to inquire the address of the data is avoided. When the number of sensors increases to cause that a new physical block is needed for additional storage and is allocated according to needs, the sensor change condition recording table also records the use condition of the internal storage space of the physical block, in which data is written before the new physical block is added, and is used for quickly inquiring the relationship between the logical page number and the offset in the logical block. In order to ensure the normal writing and reading of data when the number of the sensors changes, a sensor change condition recording table is maintained, the table records the time when the number of the sensors changes and the written data scale of the sensors when the number of the sensors changes, the content of the table is divided into a time point and logic page relation part and a logic page and logic block relation part, and on the basis of the table, the storage position of the sensor data with the specified time and the specified type can be obtained by combining a block-level address mapping table through simple calculation.
S4, inquiring the last relational parameter in the sensor change condition record table, substituting the relational parameter into an offset address obtained by calculation and writing data into a flash memory, and finishing the storage of the time series data of the Internet of things;
s5, calculating to obtain the address of the target time point data according to the relation parameter of the position of the starting time closest to the target time point in the query sensor change condition record table, and completing the retrieval of the Internet of things time series data, wherein the updating conditions of the sensor change condition record table are as follows:
when the number of the sensors is increased or decreased, updating a sensor change condition record table; when the number of sensors is reduced, the current total block number X is not updatedi(ii) a When the number of sensors is increased and a new physical block needs to be added for saving data, the current total block number X is updatedi；
The step S4 includes the following steps:
s401, retrieving data of a target time point according to the relation parameters obtained in the step S3 and the sensor change condition record table;
s402, inquiring the initial time closest to the target time point, and calculating according to the closest initial time and the relation parameters of the record list time point and the logical page to obtain a target logical page number;
the expression of the target logical page number is as follows:
wherein LPN represents the target logical page number, tnRepresents a target time point, t0Representing the last point in time before the change recorded by the sensor change table, m representing the number of sensors after the change, p representing the sampling rate of the sensors, lpnfirstThe last logical page number indicating the data deposit;
s403, comparing the target logical page with the latest logical page number recorded in the sensor change condition record table to calculate a difference value;
s404, calculating by using the difference to obtain a logic block number where the target time point data is located and offset thereof;
the expression of the logic block number of the target time point data is as follows:
lpnlasti＝lpni+Xi-lbni
wherein i represents the number of times of adding a new block to the record table, the initial value is 0, which represents the X physical block allocated to the data at the beginning, and the subsequent value is added when a physical block needs to be additionally configured for the newly added sensor, lpn represents the logical page number of the target time point, lpnlastiLogical page number indicating the last offset bit written in the old block before adding the new block lpniIndicating the logical page number to which the current data of the data was written when the block was added i, lbniIndicates the logical block number, X, in which it is locatediRepresenting the total number of blocks allocated to it after the ith addition of a block.
As shown in FIG. 3, 3 physical blocks are initially allocated for storing data, and the last offset in the round (4, 5, 6) is consistent as long as it is within a round of writing and before adding a new block, the above equation is taken into account to obtain lpn as long as any page needs to be added when writing any page in the bufferlasti＝6。
The expression of the offset of the target time point data is as follows:
wherein lpn denotes the logical page number where the target time point is located, lpnlastiLogical page number, X, representing the last offset bit written in the old block before the addition of the new blockiDenotes the total number of blocks, X, allocated to the block after the ith additionnIndicates the total number of blocks after the addition of the block n, and k indicates the number of times the block is added.
And S405, obtaining the physical address of the target data according to the logic block number of the target time point data and the offset thereof.
In this embodiment, as shown in fig. 4, fig. 4 is a data retrieval phase, and since continuous time data is written according to the size of a physical page, it is assumed that a page can store n pieces of certain data, the number of sensors is m, the sensor sampling rate is p, and the start time is t0Calculating the logical page number of the time point to be checked as
The parameters are recorded in a sensor quantity condition recording table, new parameters are added to the table when the quantity of the sensors changes, the logical page numbers corresponding to relevant time points can be quickly searched by inquiring relevant parameters, data are sequentially written backwards in each block according to time sequence during previous writing, thus the continuous logical pages written in each round can be ensured to enable the offset of data storage in each block to be equal, a new physical block is calculated from the offset, the logical page number of target data is compared with the nearest logical page number recorded in the recording table to calculate the difference, the logical block number where the target logical page number is located and the offset thereof can be calculated by using the difference, and therefore the sensor quantity condition recording table can obtain the offset of the target logical page numberThe physical address where the target data is located.
The invention establishes a plurality of mapping tables by classification, writes data into physical addresses of continuous logical address mapping in the tables sequentially, and preferentially uses physical blocks in different flash memory chips when the address mapping in each table is established, so that the parallelism characteristic of a flash memory array can be utilized to the maximum extent, the concurrent reading and writing speed of the data is effectively improved, time and logical pages in data query are recorded by a sensor change condition recording table, linear relation parameters between the logical pages and the logical blocks are recorded, the position of the data can be quickly obtained by calculation, and the overhead of additionally constructing indexes for various data is not generated The problem of low efficiency is that the data reading and writing speed is optimized in the storage based on the flash memory aiming at the time sequence data.
Claims (9)
1. A method for storing and retrieving Internet of things time sequence data of a flash memory particle array is characterized by comprising the following steps:
s1, according to the life cycle and unit time sampling rate of the stored data and the size of single time point data, allocating physical blocks for the flash memory particles in the flash memory particle array in advance from different flash memory particles, and constructing a corresponding block level address mapping table;
s2, according to the block-level address mapping table, constructing a storage unit with the size of a page in a buffer area of a memory, and writing data into the storage unit in a time sequence;
s3, creating and updating a sensor change condition recording table according to the real-time monitoring of the number and sampling rate of the sensors and the block-level address mapping table, and recording the relation parameters of the data time point and the logical page number and the relation parameters between the logical page number and the logical block number;
s4, inquiring the last relational parameter in the sensor change condition record table, substituting the relational parameter into an offset address obtained by calculation and writing data into a flash memory, and finishing the storage of the time series data of the Internet of things;
and S5, calculating to obtain the address of the target time point data according to the relation parameter of the position of the starting time closest to the target time point in the query sensor change condition record table, and completing retrieval of the time series data of the Internet of things.
2. The internet of things time series data storage and retrieval method for the flash memory particle array according to claim 1, wherein the flash memory particle array in the step S1 comprises a data type collected by a sensor, a starting time t0Start time logical page lpnfirstNumber of added blocks i, logical page lpniLogic block lbniNumber of blocks before addition Xi-1Current total number of blocks XiThe number of sensors m and the sampling rate p.
3. The internet of things time series data storage and retrieval method for the flash memory particle array according to claim 1, wherein in the step S1, each physical block is allocated in a different flash memory, each physical block only stores one type of data, and an independent block-level address mapping table is constructed for each type of data.
4. The internet of things time series data storage and retrieval method for the flash memory particle array according to claim 1, wherein the sensor change condition record table in the step S3 includes the number and the sampling rate of the current type data sensors, and the starting time and the logical page number of the starting time at the current number and the sampling rate.
5. The internet-of-things time-series data storage and retrieval method for the flash memory particle array according to claim 1, wherein the updating conditions of the sensor change condition record table in the step S3 are as follows:
updating the sensor change log table as the number of sensors increases or decreases, wherein:
when the number of sensors is reduced, the current total block number X is not updatedi；
When the number of sensors is increased and a new physical block needs to be added for saving data, the current total block number X is updatedi。
6. The internet of things time-series data storage and retrieval method for the flash memory particle array according to claim 1, wherein the step S4 comprises the steps of:
s401, retrieving data of a target time point according to the relation parameters obtained in the step S3 and the sensor change condition record table;
s402, inquiring the initial time closest to the target time point, and calculating according to the closest initial time and the relation parameters of the record list time point and the logical page to obtain a target logical page number;
s403, comparing the target logical page with the latest logical page number recorded in the sensor change condition record table to calculate a difference value;
s404, calculating by using the difference to obtain a logic block number where the target time point data is located and offset thereof;
s405, obtaining an offset address in a physical block where the target data is located according to the logical block number where the target time point data is located and the offset thereof, and completing data storage of the Internet of things time sequence based on the flash memory particle array.
7. The method for storing and retrieving internet of things time series data of a flash memory particle array according to claim 6, wherein the expression of the target logical page number in the step S402 is as follows:
wherein LPN represents the target logical page number, tnRepresents a target time point, t0The change condition of the sensor is recordedLast time point before change, m represents number of sensors after change, p represents sampling rate of sensors, lpnfirstIndicating the last logical page number of the data deposit.
8. The method for storing and retrieving internet of things time series data of a flash memory particle array as claimed in claim 6, wherein the expression of the logic block number LBN of the target time point data in the step S404 is as follows:
lpnlasti＝lpni+Xi-lbni
wherein i represents the number of times of adding a new block to the record table, the initial value is 0, which represents the X block physical block allocated to the data just after the start, and the subsequent addition when the physical block needs to be additionally configured for the newly added sensor, lpn represents the logical page number of the target time point, lpnlasti represents the logical page number of the last offset bit written in the old block before adding the new block, lpniIndicating the logical page number to which the current data of the data was written when the block was added i, lbniIndicates the logical block number, X, in which it is locatediRepresenting the total number of blocks allocated to it after the ith addition of a block.
9. The method for storing and retrieving internet of things time series data of a flash memory particle array according to claim 6, wherein the expression of the offset of the target time point data in step S404 is as follows:
where lpn denotes the logical page number at which the target point in time is located, lpnlasti denotes the logical page number of the last offset bit written in the old block before the new block is added, XiDenotes the total number of blocks, X, allocated to the block after the ith additionnIndicates the total block after the nth block additionThe number, k, indicates the number of times a block is added.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911404306.2A CN111158604B (en) | 2019-12-31 | 2019-12-31 | Internet of things time sequence data storage and retrieval method for flash memory particle array |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911404306.2A CN111158604B (en) | 2019-12-31 | 2019-12-31 | Internet of things time sequence data storage and retrieval method for flash memory particle array |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111158604A true CN111158604A (en) | 2020-05-15 |
| CN111158604B CN111158604B (en) | 2022-02-11 |
Family
ID=70559534
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911404306.2A Active CN111158604B (en) | 2019-12-31 | 2019-12-31 | Internet of things time sequence data storage and retrieval method for flash memory particle array |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111158604B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112506936A (en) * | 2021-02-04 | 2021-03-16 | 杭州罗莱迪思科技股份有限公司 | Internet of things hardware gateway data processing method and system based on single chip microcomputer |
| CN113296731A (en) * | 2021-05-25 | 2021-08-24 | 电子科技大学 | Multichannel data acquisition data caching method based on network on chip |
| CN114281248A (en) * | 2021-11-29 | 2022-04-05 | 深圳三地一芯电子有限责任公司 | Method for improving read-write performance of RAM, electronic equipment and medium |
| CN114610952A (en) * | 2022-02-28 | 2022-06-10 | 广州鼎甲计算机科技有限公司 | Effective data indexing method, system, device and storage medium |
| CN117075824A (en) * | 2023-10-17 | 2023-11-17 | 苏州元脑智能科技有限公司 | Control method and device of storage device, storage device and hard disk array card |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040088474A1 (en) * | 2002-10-30 | 2004-05-06 | Lin Jin Shin | NAND type flash memory disk device and method for detecting the logical address |
| WO2007058624A1 (en) * | 2005-11-17 | 2007-05-24 | Chee Keng Chang | A controller for non-volatile memories, and methods of operating the memory controller |
| US20110010489A1 (en) * | 2009-07-09 | 2011-01-13 | Phison Electronics Corp. | Logical block management method for a flash memory and control circuit storage system using the same |
| US20110231594A1 (en) * | 2009-08-31 | 2011-09-22 | Hitachi, Ltd. | Storage system having plurality of flash packages |
| US20120054419A1 (en) * | 2010-08-27 | 2012-03-01 | Via Technologies, Inc. | Memory device and operating method thereof |
| WO2014081719A1 (en) * | 2012-11-20 | 2014-05-30 | Peddle Charles I | Solid state drive architectures |
| US20150178001A1 (en) * | 2013-12-20 | 2015-06-25 | Silicon Motion, Inc. | Data Storage Device and Data Maintenance Method Thereof |
| US20160105642A1 (en) * | 2013-06-06 | 2016-04-14 | Tatsuya Nagase | Transmission terminal, transmission system, display method and program |
| WO2019028269A2 (en) * | 2017-08-02 | 2019-02-07 | Strong Force Iot Portfolio 2016, Llc | Methods and systems for detection in an industrial internet of things data collection environment with large data sets |
-
2019
- 2019-12-31 CN CN201911404306.2A patent/CN111158604B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040088474A1 (en) * | 2002-10-30 | 2004-05-06 | Lin Jin Shin | NAND type flash memory disk device and method for detecting the logical address |
| WO2007058624A1 (en) * | 2005-11-17 | 2007-05-24 | Chee Keng Chang | A controller for non-volatile memories, and methods of operating the memory controller |
| US20110010489A1 (en) * | 2009-07-09 | 2011-01-13 | Phison Electronics Corp. | Logical block management method for a flash memory and control circuit storage system using the same |
| US20110231594A1 (en) * | 2009-08-31 | 2011-09-22 | Hitachi, Ltd. | Storage system having plurality of flash packages |
| US20120054419A1 (en) * | 2010-08-27 | 2012-03-01 | Via Technologies, Inc. | Memory device and operating method thereof |
| WO2014081719A1 (en) * | 2012-11-20 | 2014-05-30 | Peddle Charles I | Solid state drive architectures |
| US20160105642A1 (en) * | 2013-06-06 | 2016-04-14 | Tatsuya Nagase | Transmission terminal, transmission system, display method and program |
| US20150178001A1 (en) * | 2013-12-20 | 2015-06-25 | Silicon Motion, Inc. | Data Storage Device and Data Maintenance Method Thereof |
| WO2019028269A2 (en) * | 2017-08-02 | 2019-02-07 | Strong Force Iot Portfolio 2016, Llc | Methods and systems for detection in an industrial internet of things data collection environment with large data sets |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112506936A (en) * | 2021-02-04 | 2021-03-16 | 杭州罗莱迪思科技股份有限公司 | Internet of things hardware gateway data processing method and system based on single chip microcomputer |
| CN113296731A (en) * | 2021-05-25 | 2021-08-24 | 电子科技大学 | Multichannel data acquisition data caching method based on network on chip |
| CN114281248A (en) * | 2021-11-29 | 2022-04-05 | 深圳三地一芯电子有限责任公司 | Method for improving read-write performance of RAM, electronic equipment and medium |
| CN114281248B (en) * | 2021-11-29 | 2022-12-09 | 深圳三地一芯电子有限责任公司 | Method for improving read-write performance of RAM, electronic equipment and medium |
| CN114610952A (en) * | 2022-02-28 | 2022-06-10 | 广州鼎甲计算机科技有限公司 | Effective data indexing method, system, device and storage medium |
| CN114610952B (en) * | 2022-02-28 | 2023-01-13 | 广州鼎甲计算机科技有限公司 | Effective data indexing method, system, device and storage medium |
| CN117075824A (en) * | 2023-10-17 | 2023-11-17 | 苏州元脑智能科技有限公司 | Control method and device of storage device, storage device and hard disk array card |
| CN117075824B (en) * | 2023-10-17 | 2024-02-02 | 苏州元脑智能科技有限公司 | Control method and device of storage device, storage device and hard disk array card |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111158604B (en) | 2022-02-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111158604B (en) | Internet of things time sequence data storage and retrieval method for flash memory particle array | |
| US10838859B2 (en) | Recency based victim block selection for garbage collection in a solid state device (SSD) | |
| CN102012867B (en) | Data storage system | |
| CN103631940B (en) | Data writing method and data writing system applied to HBASE database | |
| CN106502587B (en) | Hard disk data management method and hard disk control device | |
| CN101354681B (en) | Memory system, abrasion equilibrium method and apparatus of non-volatile memory | |
| CN103136121B (en) | Cache management method for solid-state disc | |
| CN109947363B (en) | Data caching method of distributed storage system | |
| US20140229656A1 (en) | Multi-Tiered Memory with Different Metadata Levels | |
| US8161240B2 (en) | Cache management | |
| CN105930282B (en) | A kind of data cache method for NAND FLASH | |
| CN109582593B (en) | FTL address mapping reading and writing method based on calculation | |
| CN112395212B (en) | Method and system for reducing garbage recovery and write amplification of key value separation storage system | |
| CN102646069A (en) | Method for prolonging service life of solid-state disk | |
| CN108121670B (en) | Mapping method for reducing solid state disk metadata back-flushing frequency | |
| CN107015763A (en) | Mix SSD management methods and device in storage system | |
| CN111026329B (en) | Key value storage system based on host management tile record disk and data processing method | |
| CN107329692A (en) | Method and storage device that a kind of data are deleted again | |
| CN112685337B (en) | Method for hierarchically caching read and write data in storage cluster | |
| CN113253926A (en) | Memory internal index construction method for improving query and memory performance of novel memory | |
| CN116737064B (en) | Data management method and system for solid state disk | |
| Fevgas et al. | A study of R-Tree performance in hybrid flash/3DXPoint storage | |
| CN104571959A (en) | Data processing method and device | |
| CN112559384B (en) | Dynamic partitioning method for hybrid solid-state disk based on nonvolatile memory | |
| Shen et al. | Overlapping Aware Zone Allocation for LSM Tree-Based Store on ZNS SSDs |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |