CN106033324A - Data storage method and device - Google Patents
Data storage method and device Download PDFInfo
- Publication number
- CN106033324A CN106033324A CN201510125757.8A CN201510125757A CN106033324A CN 106033324 A CN106033324 A CN 106033324A CN 201510125757 A CN201510125757 A CN 201510125757A CN 106033324 A CN106033324 A CN 106033324A
- Authority
- CN
- China
- Prior art keywords
- data
- time
- index
- slice
- day
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data storage method and device; the method comprises the following steps: a buffer memory data piece used for recording starting time and finish time of data pieces, wherein data of the single data piece belongs to the same period; clustering data pieces in a complete period according to the starting time and finish time so as to obtain data units; writing the data units into a storage zone; determining the finish time of the final data piece of the data unit as a time index node, and adding index information of each level time index related to the time index node and gathering information of the corresponding time period. The trivial data in the buffer memory can be clustered to obtain the data units following preset index rules; the data units are stored in the storage zone according to time sequence; each level index information corresponding to each data unit can be formed in the storage zone, thus reducing frequent access to a hard disc by trivial data single storage, and realizing fast following retrieval with high efficiency.
Description
Technical field
The present invention relates to technical field of video monitoring, particularly relate to method and the dress of the storage of a kind of data
Put.
Background technology
The often the most trifling data that data acquisition based on sequential obtains, if data acquisition time
Between the longest, the sheet number of the trifling data slice of generation is the most.In the prior art, to this number
According to storing process be by each trifling data slice write-once memory area.
As a example by the data statistics of station or the volume of the flow of passengers in market, as it is shown in figure 1, volume of the flow of passengers information
It is sent to NVR system (Network Video by IPC (IP Camra, web camera)
Recorder, network hard disk video recorder), after NVR system stores, user can be by this locality
Interface or Terminal Server Client obtain statistical result, and can be by statistical result unloading or download.
As in figure 2 it is shown, the volume of the flow of passengers information that each data slice is carried adds up the corresponding period minimum including
Time started and end time, namely T1start、T2start、…、Tnstart、Tn+1startAnd T1end、
T2end、…、Tnend；Concrete data at least include entrance number and leave number.These data
The data that sheet carries storage organization in data base is as it is shown on figure 3, each data slice is according to generation
Sequential refer to rearward data successively.By each bag data as a fragment, each fragment
Between carry out retrieving (deviation post of the next fragment of the most previous fragment storage) by side-play amount.
But storage data not merely for storage, in addition it is also necessary to data are carried out computing,
Add up, analyze or search in a particular manner.Under this storage mode, it will generation problems with:
1, hard disk is i.e. write because often receiving bag data, under the identical premise of hard-disk capacity, outside
The cycle that data produce directly affects the memory cycle, though the cycle is fixed as 15min at present, however not excluded that
Adjustment or the following IPC needing compatible third party producer in this cycle (have guest flow statistics merit
Can)；
2, the frequency writing hard disk directly depends on the cycle that external data produces, and excessively frequently may
Hard disk service life can be affected；
3, directly storing because receiving source data, its each bag data form a sheet segment information, with
15min produces bag data, within one day, will produce 96 fragments, and within 1 year, calculate with 365 days, then
Produce about 3.5 ten thousand fragments, if to add up year form to belong to all of this year by first to search out
Fragment carrying out calculates (to leaving number and entrance number adds up respectively), and recall precision is relatively low.
Summary of the invention
The method and apparatus that it is an object of the invention to provide the storage of a kind of data, it is by the buffer
The data cell of default index rule it is met, by data after trifling data being polymerized
Unit is stored in memory area chronologically, is that each data cell generates each of correspondence in memory area
Level index information, reduces the frequent access that hard disk is caused by the single storage of trifling data, deposits according to sequential
The data cell entered and the corresponding index information produced make later retrieval rapidly and efficiently.
For achieving the above object, the most by the following technical solutions:
On the one hand a kind of method that data store is used, including:
Data cached, record time started and the end time of described data slice, single described number
Same period is belonged to according to the data of sheet；
It is polymerized the data slice number in a complete cycle according to described time started and end time
According to unit；
Described data cell is write memory block；
The end time confirming last data slice of described data cell is time index node,
Add the index information of the time indexs at different levels of this time index node association and corresponding time period
Summary information.
Wherein, described according to the number in described time started and one complete cycle of end time polymerization
Data cell is obtained according to sheet, including:
If the time started of the data slice of up-to-date caching is the time started in cycle, by up-to-date caching
Data slice is newly-built for ephemeral data unit；
If the time started of the data slice of up-to-date caching is not the time started in cycle, by up-to-date caching
Data slice be aggregated to ephemeral data unit；
If the end time of the data slice of up-to-date caching is the end time in cycle, described ephemeral data
Unit is a data cell corresponding to complete cycle.
Wherein, the data in described data slice are statistical data based on sequential；
The described data slice being polymerized in a complete cycle according to described time started and end time obtains
To data cell, particularly as follows:
According to described time started and end time, by corresponding in the data slice in a complete cycle
Data accumulation obtain data cell.
Wherein, described polymerization includes at least two-stage polymerization, the corresponding described polymerization bag of described time index
Include at least two-stage time index.
Wherein, the storage file storing described data cell includes data field, index area and file header,
Described time index includes Time of Day index and month index；
Described by described data cell write memory block, particularly as follows:
According to corresponding time started, end time and the described index area record of described data cell
The storage position of current data fragment；By existing day data in described data cell write data field
Write the day data slot of interpolation after fragment, or interpolation day data slot, add and point to the day added
The index information of data slot；
The end time of last data slice of the described data cell of described confirmation is time index joint
Point, adds the index information of the time indexs at different levels of this time index node association and corresponding time
The summary information of section, including:
Confirm that end time of last data slice of described data cell is Time of Day index joint
Point, adds the summary information of all data cells in this time slice in described day data slot；
Confirm that end time of last data slice of described data cell is a month index joint
Point；Add moon data slot, described moon data slot records this month every day summary information and should
The summary information of the moon；Add the index information pointing to this month data slot；
Wherein, described index information is stored in described index area the most successively according to adding sequential；
Described day data slot and moon data slot be stored in described the most successively according to adding sequential
Data field.
Wherein, described file header record have storage the segments of file, the index position of Start Fragment,
The overall time started of file, the overall end time of file and the version number of file.
On the other hand the device that a kind of data store is used, including:
Data cache module, for data cached, records time started and the knot of described data slice
The bundle time, the data of single described data slice belong to same period；
Cycle aggregation module, for being polymerized a complete week according to described time started and end time
Data slice in phase obtains data cell；
Data write. module, for writing memory block by described data cell；
Node summarizing module, at the end of confirming last data slice of described data cell
Between be time index node, add this time index node association time indexs at different levels index letter
Breath and the summary information of corresponding time period.
Wherein, described cycle aggregation module, including:
Initial submodule, if when the beginning that the time started is the cycle of the data slice of up-to-date caching
Between, by newly-built for the data slice of up-to-date caching for ephemeral data unit；
Polymerization submodule, if the time started being used for the data slice of up-to-date caching is not the beginning in cycle
Time, the data slice of up-to-date caching is aggregated to ephemeral data unit；
Complete submodule, if at the end of the end time of the data slice of up-to-date caching is the cycle
Between, described ephemeral data unit is a data cell corresponding to complete cycle.
Wherein, the data in described data slice are statistical data based on sequential；
Described cycle aggregation module, specifically for:
According to described time started and end time, by corresponding in the data slice in a complete cycle
Data accumulation obtain data cell.
Wherein, described polymerization includes at least two-stage polymerization, the corresponding described polymerization bag of described time index
Include at least two-stage time index.
Wherein, the storage file storing described data cell includes data field, index area and file header,
Described time index includes Time of Day index and month index；
Described Data write. module, specifically for:
According to corresponding time started, end time and the described index area record of described data cell
The storage position of current data fragment；, by existing number of days in described data cell write data field
According to fragment, or write the day data slot of interpolation after adding day data slot, add to point to and add
The index information of day data slot；
Described node summarizing module, including:
First collects submodule, for confirming the end of last data slice of described data cell
Time is Time of Day index node, adds all numbers in this time slice in described day data slot
Summary information according to unit；
Second collects submodule, for confirming the end of last data slice of described data cell
Time is a month index node；Add moon data slot, described moon data slot records this moon
The summary information of every day and the summary information of this month；Add the index letter pointing to this month data slot
Breath；
Wherein, described index information is stored in described index area the most successively according to adding sequential；
Described day data slot and moon data slot be stored in described the most successively according to adding sequential
Data field.
Wherein, described file header record have storage the segments of file, the index position of Start Fragment,
The overall time started of file, the overall end time of file and the version number of file.
The beneficial effects of the present invention is: by obtaining after in the buffer trifling data being polymerized
To meeting the data cell that the index preset is regular, data cell is stored in chronologically memory area,
Memory area is each data cell generates corresponding index informations at different levels, reduces trifling data
The frequent access that hard disk is caused by single storage, the data cell being stored according to sequential and corresponding generation
Index information makes later retrieval rapidly and efficiently.
Accompanying drawing explanation
Fig. 1 is the structural representation of a kind of data acquisition storage system in prior art；
Fig. 2 is the schematic diagram of the source data of data acquisition storage system acquisition in Fig. 1；
Fig. 3 is the schematic diagram of the data storage of data acquisition storage system acquisition in Fig. 1；
Fig. 4 is the first reality of the method for a kind of data storage provided in the specific embodiment of the invention
Execute the method flow diagram of example；
Fig. 5 is the second reality of the method for a kind of data storage provided in the specific embodiment of the invention
Execute the method flow diagram of example；
Fig. 6 is the second reality of the method for a kind of data storage provided in the specific embodiment of the invention
Execute the process schematic of data storage in example；
Fig. 7 is the second reality of the method for a kind of data storage provided in the specific embodiment of the invention
Execute the process schematic of data retrieval in example；
Fig. 8 is the first reality of the device of a kind of data storage provided in the specific embodiment of the invention
Execute the block diagram of example；
Fig. 9 is the second reality of the device of a kind of data storage provided in the specific embodiment of the invention
Execute the block diagram of example.
Detailed description of the invention
For making the object, technical solutions and advantages of the present invention of greater clarity, below in conjunction with specifically
Embodiment referring to the drawings, the present invention is described in more detail.It should be understood that these describe
The most exemplary, and it is not intended to limit the scope of the present invention.Additionally, in the following description, save
Omit the description to known features and technology, to avoid unnecessarily obscuring idea of the invention.
Refer to Fig. 4, it is the side of a kind of data storage provided in the specific embodiment of the invention
The method flow diagram of the first embodiment of method.Method in the present embodiment, is mainly used in various data
Storage, is based particularly on the data of timing acquisition, such as video, audio frequency, flow of the people, vehicle flowrate
Statistics etc. data.As it can be seen, the method, including:
Step S101: data cached, records time started and the end time of described data slice,
The data of single described data slice belong to same period.
During obtaining data, external data may irregularly produce, some time period
Interior possibility is long-living much may exist multiple data slice, if by straight for multiple data slice in caching
Connect and store chronologically, memory block can be caused to there is a lot of fragmentary data, to data in statistic processes
Index and add up very inconvenient, efficiency is the lowest.
In order to realize storing in order, data being stored by the cycle, the data in each cycle can
To be made up of multiple data slice, however, to ensure that the correctness of data statistics, in each data slice
Data belong to same period.Assume with 1 hour as cycle, using integral point opening as a cycle
Time beginning, then in each data slice, the data of caching can not be across integral point, such as
The data of this time period of 00:55:08～01:03:12 are finally to cause the error of statistical result,
Because the data in this data slice can only count 01:00:00～the 01:59:59 cycle；This means
The truthful data in the data in 0:00:00～0:59:59 cycle substantially this cycle of deficiency, corresponding,
The data in 01:00:00～01:59:59 cycle are substantially beyond the truthful data in this cycle.
Step S102: be polymerized the number in a complete cycle according to described time started and end time
Data cell is obtained according to sheet.
When confirming that certain is the most all according to time started and end time in the data slice cached
When data corresponding to phase have cached, the data slice in this cycle is aggregated into an entirety number
According to unit, concrete polymerization methods is polymerized according to different data types, such as video, sound
Frequency or numerical value.
Step S103: described data cell is write memory block.
Memory block refers in storage device for storing one piece of physical region of data, data cell is write
Enter in the process prior art of memory block and realized, be not described further at this.
Step S104: the end time confirming last data slice of described data cell is the time
Index node, adds index information and the correspondence of the time indexs at different levels of this time index node association
The summary information of time period.
This programme is directed to the data carrying out storing chronologically, when the indexed mode of its correspondence is also
Sequence index.The end time of last data slice according to each data cell is it may determine that go out
Whether it is the node generating index information, if it has, directly generate its correspondence according to index type
The related content in index region.
In sum, default by being met after in the buffer trifling data being polymerized
The data cell of index rule, is stored in memory area chronologically, in memory area by data cell
Generate corresponding index informations at different levels for each data cell, reduce the single storage of trifling data to firmly
The frequent access that dish causes, the data cell being stored according to sequential and the corresponding index information produced make
Later retrieval is rapidly and efficiently.
Refer to Fig. 5, it is the method for a kind of data storage provided in invention detailed description of the invention
The method flow diagram of the second embodiment, as it can be seen, the method includes:
Step S201: data cached, records time started and the end time of described data slice,
The data of single described data slice belong to same period.
It is said that in general, a length of 1 hour of cycle corresponding to each data cell.For great majority
Data statistics for, within 1 hour, be a relatively common elementary statistics cycle, but in difference
Application, different to the precision of data statistics, so 1 hour simply preferred cycle
Length, can arrange the more accurate cycle, such as 30 minutes, 10 minutes, 1 minute etc..
Step S202: if the time started of the data slice of up-to-date caching is the time started in cycle, will
The data slice of up-to-date caching is newly-built for ephemeral data unit.
Step S203: if the time started of the data slice of up-to-date caching is not the time started in cycle,
The data slice of up-to-date caching is aggregated to ephemeral data unit.
Step S204: if the end time of the data slice of up-to-date caching is the end time in cycle, institute
State ephemeral data unit and be a data cell corresponding to complete cycle.
The data slice of up-to-date caching is according to time started and the judgement of end time, corresponding different place
Reason mode, will judge for each data slice, time started and end time, finally
Need response is above-mentioned three kinds of situations.The first situation data slice is first number in a cycle
According to sheet, by newly-built for this data slice for ephemeral data unit；The second situation data slice is not a week
First data slice of phase, then be aggregated to ephemeral data unit by this data slice；The third situation number
It is last data slice in a cycle according to sheet, then this data slice is aggregated to ephemeral data unit
After, ephemeral data unit is exactly a data cell corresponding to complete cycle.For individual data sheet
For be not the situation that one of them only can occur, if the length of individual data sheet is just one
The length in individual cycle, then the first situation and the third situation all can occur for this data slice；
If individual data sheet is just last data slice in cycle, then the second situation and the 3rd
The situation of kind all can occur for this data slice.The time started of data slice and end time and one
The time started in cycle and the judgement of end time are fairly simple.Such as using 1 hour as the cycle,
Using integral point as the time started in a cycle, then the period in a cycle is
Aa:00:00～aa:59:59, corresponding, the time started is that the data slice of aa:00:00 is this cycle
First data slice, the end time is last number that the data slice of aa:59:59 is this cycle
According to sheet.
For the data of general numerical value class based on sequential, the process of polymerization is exactly in fact one and tires out
The process added, according to described time started and end time, by the data slice in a complete cycle
The data accumulation of middle correspondence obtains data cell.
Step S205: according to time started, end time and described rope that described data cell is corresponding
Draw the storage position of the current data fragment of district's record；By described data cell write data field in
Some days data slot, or write the day data slot of interpolation after adding day data slot, interpolation refers to
To add day data slot index information.
The storage file storing described data cell includes data field, index area and file header.Described
Polymerization includes at least two-stage polymerization, and the corresponding described polymerization of described time index includes at least two-stage time
Index, such as, be set to Time of Day index and month index two-stage.Described file header record is deposited
The storage segments of file, the index position of Start Fragment, overall time started of file, file
Overall end time and the version number of file.
Step S206: confirm when end time of last data slice of described data cell is day
Between index node, in described day data slot, add the remittance of all data cells in this time slice
Total information.
In order to realize express statistic and retrieval, for the statistical time range of each routine, in this statistics
At the end of period, the most first generating the basic summary information of this statistical time range, subordinate collects letter substantially
Breath is also the basic element of data storage in higher level's time slice.In this programme, with 1 hour it was
One cycle, it is however generally that, then the statistical time range of upper level is exactly sky, so in day data slot
The data of storage, except data corresponding to the data cell in 24 cycles, also include this 24 number
The summary information that rough estimates obtain is carried out according to unit.
Step S207: confirm when end time of last data slice of described data cell is month
Between index node；Add moon data slot, described moon data slot records collecting of this month every day
Information and the summary information of this month；Add the index information pointing to this month data slot.
Described index information is stored in described index area the most successively according to adding sequential；Described
Day data slot and moon data slot are stored in described data the most successively according to adding sequential
District.
Summary information in each day data slot is extracted the number in moon data slot
According to content, correspondence, the last day of every month also can produce a moon summary information, is believed by index
Breath can realize the quick location of all data, retrieves.
Refer to Fig. 6, it is the process schematic of data storage in the present embodiment, and it is based on artificial abortion
Amount data realize, and the basic parameter of statistics is entrance number and leaves number, the cycle of data cell
It it is 1 hour.As it can be seen, in concrete statistic processes, external data may irregularly be produced
Raw, may produce multiple in some time period, thus cause caching there may be such as Fig. 6
The multiple data slice shown in the top, if these data slice are used mode as shown in Figure 3
Storage, can cause having in hard disk a lot of fragmentary data the most in order, if needing to retrieve form, needs
All of data slice is retrieved and added up, inefficient, directly affect Consumer's Experience.
The scheme used in the present invention is first the data slice in each cycle to be carried out based on the default cycle
Polymerization, specific to Fig. 6 be exactly by a hour interior data slice record entrance number and from
The number of opening is separately summed the data that polymerization obtains in data cell, record in data slice and data cell
Really same class data, the most corresponding time range is different.
By above-mentioned polymerization, the data belonging to same hour have only to a data cell preservation,
Thus reached the purpose of data compression, decrease fragmentation of data, control the quantity of fragment index.
If aggregation information by the hour, within one day, produce 24 data cells.If entered according to data cell
Row storage, one day 24 data cell, if 1 year by 365 days statistics, then will record 8760
Individual data cell.Retrieve with year form, it is necessary first to retrieve 8760 data cells, so
After also need to these data cells are calculated, finally obtain the summary information of this year every month.
Under this storage mode, recall precision increases than the scheme in Fig. 3, but still does not reaches
The requirement of real-time retrieval, for improving recall precision further.Horal data cell is carried out
After polymerization obtains a day fragment；As shown in Figure 6, the data in each data cell are stored in day data
In fragment, a complete day data slot includes 24 data cells and a summary information,
A hour unit information on the same day condenses together, and forms day data slot, and day data slot is made
For the container of data cell on the same day, it is responsible for this day all data cell of storage, by data cell daily
File.Store with upper type, if 1 year by 365 days statistics, then will record for 365 day
For fragment, relatively 8760 fragments, retrieval hit rate has had the lifting of more than 20 times.Base
In Report form application scene, when detecting real time data, at the end of every day, this sky is owned
Data cell carry out collecting the summary information obtaining every day, and tie in the last day of each month
During bundle, a moon data slot is added up and be aggregated into the summary information to this month every day.Month data slice
Section is not the set of day data slot, but records the day summary information of this month every day one by one, and
Obtain a moon summary information based on all of day summary information, count real-time dynamicly when Data Detection
Calculate, this avoid when retrieval, the calculating to data, on embedded platform, especially calculate ratio
Relatively take cpu resource.In figure 6, it illustrates that the moon data slot of January, then
Data slot is comprising by the moon of January the day summary information and this month of every day 31 day this month
Month summary information.
The record of data is typically to embody in the way of file, in order to ensure the privacy of data,
When day data slot and moon data slot are stored in memory block, this programme have employed specific file
Form.
As shown in Figure 6, file comprises data field, index area and file header.Data field is mainly used
In storing concrete data message, data slot is divided into day data slot and moon data slot, every day
To produce day data slot, each the end of month will produce a month data slot, herein data
Fragment is the set of data, will belong on the same day hour data cell polymerization formed day data
Fragment, forms moon data slot it can be understood as be by the day summary information polymerization belonging to same January
A kind of filing form of information.
Index area is for storing the information, i.e. fragment such as the concrete position of each data slot of data field
Index, itself and concrete data slot one_to_one corresponding, record one day data slot the most simultaneously can record
One day fragment index, for position correspondence day data slot；Record a month data slot then
A month fragment index can be recorded, for positioning the moon data slot of correspondence, for data slice simultaneously
The retrieval of section, essence is the retrieval to fragment index, then carries out data locking by fragment index,
Can be understood as fragment index is catalogue, and sheet segment information is then data slot particular content.
File header is mainly used in the version number of log file, the establishment time of file, fragment index number
Deng.File header is stored in the fixed position of file end, and index area is reverse by file header end position
Storage, data field forward from front to back stores.
Data cell in single day data slot is indexed by information offset amount, the most previous
Data cell records next number information offset amount according to unit, it is achieved between concrete data cell
Location and Information expansion.In month fragment data, the summary information of this month every day of record uses equally
The mode of information offset amount carries out forward-backward correlation.
By above-mentioned storage mode, in terms of the calculating two of the hit rate and reduction retrieval that improve retrieval
Improve the efficiency of data retrieval, thus meet under embedded platform data retrieval real-time
Property.
Based on above-mentioned data buffer storage, polymerization and memory technology, data store based on sequential.With
As a example by the information of 1 year, the most only 365 days data slots of the quantity of data slot and 12
Month data slot, compares 3.5 ten thousand fragments of former scheme, only number of fragments and compresses about 92 times.And
And logical relation between each fragment is clear, the retrieval for information substantially increases hit rate,
Statistics for data seldom has extra calculating.
Such as need to search for the data in January, 2014, first read the relevant letter of file at file header
Breath judges that data to be searched for, whether at this document, if it is search the rope in January in index area
Fuse ceases, and by the moon data slot in January in data field, index information location, its order is read,
The summary information of every day in January can be obtained.If needing the data specific to every day, then lead to
Crossing the index information of this day can realize quickly positioning, position fixing process is as shown in Figure 7.
Of that month fragment does not exists (under abnormal conditions), and fast search will be switched to degradation search, right
Belong to all of day data slot of this month to be read out, if as it is shown in fig. 7, the moon sheet in January
Segment data does not exists, then read the day data slot of every day, day according to the index information of every day
Fragment data has the summary information of every day, the summary information of every day is carried out cumulative
Obtain the summary information of 31 days this months.
Based on above-mentioned day data slot and moon data slot, can derive further daily sheet,
Weekly return, moon form and year form, various forms select corresponding day data slot or moon data slice
Duan Jinhang adds up.
In the present embodiment, for convenience of description, use Time of Day index and Time of Day indexes
Two-stage index mode, in actual application, for concrete application scenarios demand, permissible
Other time index mode, such as day, ten days, the three level list mode in season are set；To data
Processing procedure also can be for the data record etc. in scientific experiments.
In sum, default by being met after in the buffer trifling data being polymerized
The data cell of index rule, is stored in memory area chronologically, in memory area by data cell
Generate corresponding index informations at different levels for each data cell, reduce the single storage of trifling data to firmly
The frequent access that dish causes, the data cell being stored according to sequential and the corresponding index information produced make
Later retrieval is rapidly and efficiently.The file format of independent design ensure that the quick storage of data, reduces
The waste of memory space, enhances the privacy of data simultaneously.
The enforcement of the device of a kind of data storage provided in the specific embodiment of the invention is provided
Example, the embodiment of the embodiment of the device of data storage method based on above-mentioned data storage is real
Existing, the most most in the embodiment of the device of data storage elaboration, refer to above-mentioned data storage
The embodiment of method.
Refer to Fig. 8, it is the dress of a kind of data storage provided in the specific embodiment of the invention
The block diagram of the first embodiment put, as it can be seen, this device, including:
Data cache module 310, for data cached, records the time started of described data slice
And the end time, the data of single described data slice belong to same period；
Cycle aggregation module 320, for complete according to described time started and end time polymerization one
Data slice in complete cycle obtains data cell；
Data write. module 330, for writing memory block by described data cell；
Node summarizing module 340, for confirming the knot of last data slice of described data cell
The bundle time is time index node, adds the rope of the time indexs at different levels of this time index node association
Fuse breath and the summary information of corresponding time period.
Design in order to ensure this device effectively realizes, especially in embedded platform, from hardware cell
For the angle of frame, for the above-mentioned each module in bogey, reduce the load to system, protect
Card data process and the real-time of storage.The system framework of hardware uses role's division of labor, mainly includes
Listener, distributor and executor.Listener monitors the number from data acquisition end (such as IPC)
According to sheet, and data slice is inserted the source information chained list cached, automatically compression data fixed cycle write
Reduced data chained list, waits that distributor distributes.Distributor observes reduced data chained list, if
There are the data being suitable for write, then start executor and perform the write of data.Executor is responsible for storage
District's write data.
It is said that in general, because NVR system resource is the most nervous, listener and distributor distinguish
Start 1 thread, and executor can start multiple thread and quickly perform what distributor distributed
Order.
By above technical scheme, in the case of embedded platform resource-constrained, data are used to delay
The method deposited and be polymerized, is achieved in rational system framework, and respectively with day, week, the moon,
Year form, carries out actual retrieval test, and recall precision has reached the response of intended second level.
In sum, the collaborative work of above-mentioned each unit, by entering trifling data in the buffer
It is met the data cell of default index rule after row polymerization, data cell is stored in chronologically
Memory area, is that each data cell generates corresponding index informations at different levels in memory area, subtracts
The frequent access that hard disk is caused by few single storage of trifling data, the data cell being stored according to sequential and
The corresponding index information produced makes later retrieval rapidly and efficiently.
Refer to Fig. 9, it is the dress of a kind of data storage provided in the specific embodiment of the invention
The block diagram of the second embodiment put, as it can be seen, this device, including:
Data cache module 310, for data cached, records the time started of described data slice
And the end time, the data of single described data slice belong to same period；
Cycle aggregation module 320, for complete according to described time started and end time polymerization one
Data slice in complete cycle obtains data cell；
Data write. module 330, for writing memory block by described data cell；
Node summarizing module 340, for confirming the knot of last data slice of described data cell
The bundle time is time index node, adds the rope of the time indexs at different levels of this time index node association
Fuse breath and the summary information of corresponding time period.
Wherein, described cycle aggregation module 320, including:
Initial submodule 321, if being opening of cycle for the time started of the data slice of up-to-date caching
Time beginning, by newly-built for the data slice of up-to-date caching for ephemeral data unit；
Polymerization submodule 322, if not being the cycle for the time started of the data slice of up-to-date caching
Time started, the data slice of up-to-date caching is aggregated to ephemeral data unit；
Complete submodule 323, if the knot that the end time is the cycle of the data slice for up-to-date caching
The bundle time, described ephemeral data unit is a data cell corresponding to complete cycle.
Wherein, the data in described data slice are statistical data based on sequential；
Described cycle aggregation module 320, specifically for:
According to described time started and end time, by corresponding in the data slice in a complete cycle
Data accumulation obtain data cell.
Wherein, described polymerization includes at least two-stage polymerization, the corresponding described polymerization bag of described time index
Include at least two-stage time index.
Wherein, the storage file storing described data cell includes data field, index area and file header,
Described time index includes Time of Day index and month index；
Described Data write. module 330, specifically for:
According to corresponding time started, end time and the described index area record of described data cell
The storage position of current data fragment；, by existing number of days in described data cell write data field
According to fragment, or write the day data slot of interpolation after adding day data slot, add to point to and add
The index information of day data slot；
Described node summarizing module 340, including:
First collects submodule 341, for confirming last data slice of described data cell
End time is Time of Day index node, adds institute in this time slice in described day data slot
There is the summary information of data cell；
Second collects submodule 342, for confirming last data slice of described data cell
End time is a month index node；Add moon data slot, record in described moon data slot
The summary information of this month every day and the summary information of this month；Add the index pointing to this month data slot
Information；
Wherein, described index information is stored in described index area the most successively according to adding sequential；
Described day data slot and moon data slot be stored in described the most successively according to adding sequential
Data field.
Wherein, described file header record have storage the segments of file, the index position of Start Fragment,
The overall time started of file, the overall end time of file and the version number of file.
In sum, the cooperative cooperating of above-mentioned functions module, by the buffer by trifling data
It is met the data cell of default index rule after being polymerized, data cell is deposited chronologically
Enter memory area, be that each data cell generates corresponding index informations at different levels in memory area,
Reduce the frequent access that hard disk is caused by the single storage of trifling data, the data cell being stored according to sequential
The index information produced with correspondence makes later retrieval rapidly and efficiently.The file format of independent design is protected
Demonstrate,prove the quick storage of data, decreased the waste of memory space, enhance the secret of data simultaneously
Property.
It should be appreciated that the above-mentioned detailed description of the invention of the present invention be used only for exemplary illustration or
Explain the principle of the present invention, and be not construed as limiting the invention.Therefore, without departing from the present invention
Spirit and scope in the case of any modification, equivalent substitution and improvement etc. done, all should comprise
Within protection scope of the present invention.Additionally, claims of the present invention are intended to fall into institute
Whole changes in the equivalents on attached right and border or this scope and border
And modification.
Although embodiments of the present invention are described in detail it should be understood that, not
In the case of deviation the spirit and scope of the present invention, embodiments of the present invention can be made various
Change, replace and change.
Claims (12)
1. the method for a data storage, it is characterised in that including:
Data cached, record time started and the end time of described data slice, single described
The data of data slice belong to same period；
Obtain according to the data slice that described time started and end time are polymerized in a complete cycle
To data cell；
Described data cell is write memory block；
Confirm that end time of last data slice of described data cell is time index joint
Point, add the index information of the time indexs at different levels of this time index node association and corresponding time
Between the summary information of section.
The method of a kind of data the most according to claim 1 storage, it is characterised in that institute
State and obtain according to the data slice in described time started and one complete cycle of end time polymerization
Data cell, including:
If the time started of the data slice of up-to-date caching is the time started in cycle, by up-to-date caching
Data slice newly-built for ephemeral data unit；
If the time started of the data slice of up-to-date caching is not the time started in cycle, by up-to-date slow
The data slice deposited is aggregated to ephemeral data unit；
If the end time of the data slice of up-to-date caching is the end time in cycle, described nonce
It is a data cell corresponding to complete cycle according to unit.
The method of a kind of data the most according to claim 1 storage, it is characterised in that institute
Stating the data in data slice is statistical data based on sequential；
Described according to the data in described time started and one complete cycle of end time polymerization
Sheet obtains data cell, particularly as follows:
According to described time started and end time, by right in the data slice in a complete cycle
The data accumulation answered obtains data cell.
The method of a kind of data the most according to claim 1 storage, it is characterised in that institute
Stating polymerization and include at least two-stage polymerization, the corresponding described polymerization of described time index includes at least two-stage
Time index.
The method of a kind of data the most according to claim 4 storage, it is characterised in that deposit
The storage file storing up described data cell includes data field, index area and file header, described time
Index includes Time of Day index and month index；
Described by described data cell write memory block, particularly as follows:
According to time started, end time and described index area record that described data cell is corresponding
The storage position of current data fragment；By existing day in described data cell write data field
Write the day data slot of interpolation after data slot, or interpolation day data slot, add sensing and add
Add day data slot index information；
The end time of last data slice of the described data cell of described confirmation is time rope
Draw node, add index information and the correspondence of the time indexs at different levels of this time index node association
The summary information of time period, including:
Confirm that end time of last data slice of described data cell is Time of Day index
Node, that adds all data cells in this time slice in described day data slot collects letter
Breath；
Confirm that end time of last data slice of described data cell is a month index
Node；Add moon data slot, described moon data slot records the summary information of this month every day
Summary information with this month；Add the index information pointing to this month data slot；
Wherein, described index information is stored in described index the most successively according to adding sequential
District；Described day data slot and moon data slot be stored in the most successively according to adding sequential
Described data field.
The method of a kind of data the most according to claim 5 storage, it is characterised in that institute
State file header record have storage the segments of file, the index position of Start Fragment, file whole
Body time started, the overall end time of file and the version number of file.
7. the device of a data storage, it is characterised in that including:
Data cache module, for data cached, record described data slice time started and
End time, the data of single described data slice belong to same period；
Cycle aggregation module, for being polymerized one completely according to described time started and end time
Data slice in cycle obtains data cell；
Data write. module, for writing memory block by described data cell；
Node summarizing module, for confirming the end of last data slice of described data cell
Time is time index node, adds the rope of the time indexs at different levels of this time index node association
Fuse breath and the summary information of corresponding time period.
The device of a kind of data the most according to claim 7 storage, it is characterised in that institute
State cycle aggregation module, including:
Initial submodule, if the beginning that the time started is the cycle of the data slice for up-to-date caching
Time, by newly-built for the data slice of up-to-date caching for ephemeral data unit；
Polymerization submodule, if time started not the opening for the cycle of the data slice for up-to-date caching
Time beginning, the data slice of up-to-date caching is aggregated to ephemeral data unit；
Complete submodule, if the end that the end time is the cycle of the data slice for up-to-date caching
Time, described ephemeral data unit is a data cell corresponding to complete cycle.
The device of a kind of data the most according to claim 7 storage, it is characterised in that institute
Stating the data in data slice is statistical data based on sequential；
Described cycle aggregation module, specifically for:
According to described time started and end time, by right in the data slice in a complete cycle
The data accumulation answered obtains data cell.
The device of a kind of data the most according to claim 7 storage, it is characterised in that institute
Stating polymerization and include at least two-stage polymerization, the corresponding described polymerization of described time index includes at least two-stage
Time index.
The device of 11. a kind of data according to claim 10 storages, it is characterised in that
The storage file storing described data cell includes data field, index area and file header, time described
Between index include Time of Day index and month index；
Described Data write. module, specifically for:
According to time started, end time and described index area record that described data cell is corresponding
The storage position of current data fragment；By existing day in described data cell write data field
Write the day data slot of interpolation after data slot, or interpolation day data slot, add sensing and add
Add day data slot index information；
Described node summarizing module, including:
First collects submodule, for confirming the knot of last data slice of described data cell
The bundle time is Time of Day index node, adds institute in this time slice in described day data slot
There is the summary information of data cell；
Second collects submodule, for confirming the knot of last data slice of described data cell
The bundle time is a month index node；Add moon data slot, record in described moon data slot
The summary information of this month every day and the summary information of this month；Add the rope pointing to this month data slot
Fuse ceases；
Wherein, described index information is stored in described index the most successively according to adding sequential
District；Described day data slot and moon data slot be stored in the most successively according to adding sequential
Described data field.
The device of 12. a kind of data according to claim 11 storages, it is characterised in that
Described file header record has the storage segments of file, the index position of Start Fragment, file
Overall time started, the overall end time of file and the version number of file.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2015101028390 | 2015-03-09 | ||
| CN201510102839 | 2015-03-09 |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106033324A true CN106033324A (en) | 2016-10-19 |
| CN106033324B CN106033324B (en) | 2020-03-06 |
Family
ID=57149126
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510125757.8A Active CN106033324B (en) | 2015-03-09 | 2015-03-20 | Data storage method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106033324B (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106998436A (en) * | 2017-03-24 | 2017-08-01 | 浙江宇视科技有限公司 | Video recording backup method and device |
| CN108183904A (en) * | 2017-12-29 | 2018-06-19 | 金华芒果信息技术有限公司 | Mobile data method for real-time monitoring, construction system and device |
| CN108228479A (en) * | 2018-01-29 | 2018-06-29 | 深圳市泰比特科技有限公司 | A kind of embedded flash data storage method and system |
| CN108345427A (en) * | 2017-01-25 | 2018-07-31 | 杭州海康威视数字技术股份有限公司 | A kind of hard disc data storage method and device |
| CN108446399A (en) * | 2018-03-29 | 2018-08-24 | 重庆大学 | A kind of dynamic memory optimization method of structuring magnanimity real time data |
| CN112214493A (en) * | 2020-10-20 | 2021-01-12 | 北京九思易自动化软件有限公司 | Historical record storage method and device, electronic equipment and storage medium |
| CN112650756A (en) * | 2020-12-29 | 2021-04-13 | 成都科来网络技术有限公司 | Time projection indexing method and system based on time sequence data |
| CN113448997A (en) * | 2021-06-22 | 2021-09-28 | 深信服科技股份有限公司 | Cache processing method and device, electronic equipment and storage medium |
| CN113468187A (en) * | 2021-09-02 | 2021-10-01 | 太平金融科技服务(上海)有限公司深圳分公司 | Multi-party data integration method and device, computer equipment and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030110352A1 (en) * | 2001-12-12 | 2003-06-12 | Fujitsu Limited | Data storage apparatus that combines a group of continuous data series stored in buffer separately before storing in a recording medium |
| CN101354636A (en) * | 2008-09-08 | 2009-01-28 | 创新科存储技术（深圳）有限公司 | Method and system for writing data into magnetic disk array |
| CN103607463A (en) * | 2013-11-25 | 2014-02-26 | 中国电信集团系统集成有限责任公司 | Positioning data storage system and method |
-
2015
- 2015-03-20 CN CN201510125757.8A patent/CN106033324B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030110352A1 (en) * | 2001-12-12 | 2003-06-12 | Fujitsu Limited | Data storage apparatus that combines a group of continuous data series stored in buffer separately before storing in a recording medium |
| CN101354636A (en) * | 2008-09-08 | 2009-01-28 | 创新科存储技术（深圳）有限公司 | Method and system for writing data into magnetic disk array |
| CN103607463A (en) * | 2013-11-25 | 2014-02-26 | 中国电信集团系统集成有限责任公司 | Positioning data storage system and method |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108345427B (en) * | 2017-01-25 | 2020-09-04 | 杭州海康威视数字技术股份有限公司 | Hard disk data storage method and device |
| CN108345427A (en) * | 2017-01-25 | 2018-07-31 | 杭州海康威视数字技术股份有限公司 | A kind of hard disc data storage method and device |
| CN106998436A (en) * | 2017-03-24 | 2017-08-01 | 浙江宇视科技有限公司 | Video recording backup method and device |
| CN106998436B (en) * | 2017-03-24 | 2020-09-01 | 浙江宇视科技有限公司 | Video backup method and device |
| CN108183904B (en) * | 2017-12-29 | 2020-12-01 | 北京网高科技股份有限公司 | Mobile data real-time monitoring method, framework system and device |
| CN108183904A (en) * | 2017-12-29 | 2018-06-19 | 金华芒果信息技术有限公司 | Mobile data method for real-time monitoring, construction system and device |
| CN108228479A (en) * | 2018-01-29 | 2018-06-29 | 深圳市泰比特科技有限公司 | A kind of embedded flash data storage method and system |
| CN108446399A (en) * | 2018-03-29 | 2018-08-24 | 重庆大学 | A kind of dynamic memory optimization method of structuring magnanimity real time data |
| CN108446399B (en) * | 2018-03-29 | 2021-07-30 | 重庆大学 | Dynamic storage optimization method for structured massive real-time data |
| CN112214493A (en) * | 2020-10-20 | 2021-01-12 | 北京九思易自动化软件有限公司 | Historical record storage method and device, electronic equipment and storage medium |
| CN112650756A (en) * | 2020-12-29 | 2021-04-13 | 成都科来网络技术有限公司 | Time projection indexing method and system based on time sequence data |
| CN113448997A (en) * | 2021-06-22 | 2021-09-28 | 深信服科技股份有限公司 | Cache processing method and device, electronic equipment and storage medium |
| CN113468187A (en) * | 2021-09-02 | 2021-10-01 | 太平金融科技服务(上海)有限公司深圳分公司 | Multi-party data integration method and device, computer equipment and storage medium |
| CN113468187B (en) * | 2021-09-02 | 2021-11-23 | 太平金融科技服务(上海)有限公司深圳分公司 | Multi-party data integration method and device, computer equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106033324B (en) | 2020-03-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106033324A (en) | Data storage method and device | |
| US10064136B2 (en) | Method and device for pushing information | |
| CN103593376B (en) | A kind of method and device for gathering user behavior data | |
| US9934233B2 (en) | Searchable data archive | |
| CN101216821B (en) | Data acquisition system storage management method | |
| CN102065283B (en) | Storage management method and device of video monitoring data | |
| US20210333994A1 (en) | Method for aggregation optimization of time series data | |
| CN104112010B (en) | A kind of date storage method and device | |
| CN106599292B (en) | Method and system for storing real-time video data and image data | |
| CN104427519B (en) | IP address ownership place management method and device | |
| CN111371672B (en) | Message pushing method and device | |
| US20220075791A1 (en) | Storing data items and identifying stored data items | |
| CN107491549A (en) | A kind of data processing method and system | |
| CN103678314B (en) | Mass data processing system, the device and method extracted based on correlation rule | |
| CN103220587B (en) | A kind of method and device obtaining time-shifted contents | |
| CN108280123A (en) | A kind of row polymerization of HBase | |
| CN110765221A (en) | Management method and device of space-time trajectory data | |
| CN111913913A (en) | Access request processing method and device | |
| CN116881277A (en) | Data aggregation method, apparatus and computer readable medium | |
| CN112804566A (en) | Program recommendation method, device and computer readable storage medium | |
| US10956369B1 (en) | Data aggregations in a distributed environment | |
| CN104424236B (en) | Storage method and device in data acquisition | |
| CN106998436B (en) | Video backup method and device | |
| CN107025266B (en) | Service data processing method and device | |
| CN110990239A (en) | System and method for collecting user data in user equipment based on soft sensor |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |