CN103793468A - Data storage method and device and data reading method and device - Google Patents
Data storage method and device and data reading method and device Download PDFInfo
- Publication number
- CN103793468A CN103793468A CN201310741482.1A CN201310741482A CN103793468A CN 103793468 A CN103793468 A CN 103793468A CN 201310741482 A CN201310741482 A CN 201310741482A CN 103793468 A CN103793468 A CN 103793468A
- Authority
- CN
- China
- Prior art keywords
- data
- data storage
- mark
- area
- primary importance
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000013500 data storage Methods 0.000 title claims abstract description 69
- 238000000034 method Methods 0.000 title claims abstract description 63
- 238000012163 sequencing technique Methods 0.000 claims description 4
- 238000010586 diagram Methods 0.000 description 6
- PEDCQBHIVMGVHV-UHFFFAOYSA-N Glycerine Chemical compound OCC(O)CO PEDCQBHIVMGVHV-UHFFFAOYSA-N 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000003780 insertion Methods 0.000 description 1
- 230000037431 insertion Effects 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0643—Management of files
Abstract
The invention discloses data storage method and device and data reading method and device. The data storage method includes a), appointing identifiers for to-be-stored data, and b), storing the to-be-stored data at a first position in a first region of a data storage file, and storing the identifiers and additional information related with the to-be-stored data at a second position in a second region of the data storage file. By the data storage method and device and the data reading method and device, problems about access speed, limitation in storage format and system resource occupation in the prior art are solved, and good technical effect is achieved.
Description
Technical field
The present invention relates to data storage and read field, being specifically related to a kind of date storage method and device and method for reading data and device.
Background technology
In existing application system, mostly relate to Data Access Technology, comprise that data storage and data read.Data are stored on hardware device, and application program needs constantly mutual with hardware device, realize the storage of data and read, a small amount of data storage and reading, general processing time that all need to be extremely short.
Along with improving constantly of the level of informatization, the memory data output of application system also constantly promotes, data storage capacities and data reading capability seem even more important in application system, and it can determine the performance of application system to a great extent, also may become performance index of application system.In the high application of some requirement of real-time, also may affect integrality, real-time, validity of storage etc.
In modern Application system, the factor that memory technology must be considered comprises: the storage of (1) high speed event context.High speed event context produces the environment of a large amount of events fast, in 10,000,000,000 networks, produces a large amount of network events at a high speed, to the storage of these network events.In high speed event context, the memory technology that have, the mass data storage rapidly these moments being produced, integrality and the real-time of assurance data; (2) mass data quick-searching.Data retrieval is access technique inalienable part, for the mass data of storing in high speed event context, need to have inquiry mechanism faster, guarantees the rapid extraction of desired data; (3) system resources consumption.The total resource of system is as always limited in CPU, internal memory etc., and application system itself also can consume system resources, and therefore mass data is in storage with while reading, and the mechanism that should have should not consume too many system resource.
Relation data library storage is conventional data storage technology, and kinds of relationships database all has more general interface, and open office terminal and management method is provided.Relational database memory technology can reach expection storage effect in low speed event context, but in high speed event context, database storage techniques will be very limited.Specifically, its shortcoming is: (1) storage speed is slow, the data insertion speed peak value of database storage techniques can approach 10,000 per second, in high speed event context, in the time that the event producing reaches up to ten thousand per second, database storage techniques will be difficult to meet the demands; (2) storage data layout is limited, and the data layout that database storage techniques is stored it has considerable restraint, depends on the technology that realizes of database itself; (3) occupying system resources is larger, the necessary log-on data of database storage techniques storehouse server, and this also can take larger system resource.
Summary of the invention
For the above-mentioned defect existing in prior art, the present invention proposes following date storage method and device and method for reading data and device.
The present invention proposes a kind of date storage method.The method comprises: a) the data assigned identification for storing; B) the described data that will store of the primary importance storage in the first area of data storage file, and the second place in the second area of described data storage file is stored described mark and the additional information relevant with the described data that will store.
The invention allows for a kind of data storage device.This device comprises: mark assignment module, is used to the data assigned identification that will store; Data memory module, stores described mark and the additional information relevant with the described data that will store for the described data that will store of primary importance storage of the first area at data storage file and the second place in the second area of described data storage file.
The invention allows for a kind of method for reading data.The method comprises: in the described second area of described data storage file, look for the mark that the data that will read are assigned; Determine the primary importance of the described data that will read in the described first area of described data storage file according to the described additional information of storing together with described mark; And the data that will read described in reading.
The invention allows for a kind of data fetch device, this device comprises: mark is searched module, the mark of assigning for look for the data that will read at the described second area of described data storage file; Position determination module, for determining the primary importance of the described data that will read in the described first area of described data storage file according to the described additional information of storing together with described mark; And data read module, for reading the described data that will read.
By foregoing, the present invention, by specific identifier is combined to storage in data access process with data, has solved the access speed problem, storage format restricted problem and the occupying system resources problem that in prior art, exist, has realized good technique effect.
Accompanying drawing explanation
Fig. 1 shows the process flow diagram of the date storage method proposing according to one embodiment of the present invention;
Fig. 2 shows the process flow diagram of the method for reading data proposing according to one embodiment of the present invention; And
Fig. 3 shows according to the schematic diagram of the file layout of the data storage file of one embodiment of the present invention.
Embodiment
Fig. 1 shows the process flow diagram of date storage method proposed by the invention.
In step 110, the first data assigned identification for storing.In the time of storage data, in file, store a data block that length is known, and assign unique mark (ID) to these data at every turn.
In embodiment preferably, can assign and there is the ID of equal length and can increase progressively according to the time sequencing of assigning for all data, the ID of new data block is larger than the ID of old data block.
For example, ID can have following form:
ID=event rise time+N position overall situation sequential index/Nmax.
Wherein:
The event rise time: by a string character representation that is accurate to microsecond, (2) point (2) second (2) microseconds (5) while comprising the day moon in year (2) (2) (2), as 13062417093023121
N position overall situation sequential index: one by 1,2,3 ... the character string that natural number forms;
Nmax: maximum index value (as 10000).
Typical ID character string can be for example: 1306241709302312100001,1306241709302312100002,130624170930231210003 etc.
It should be appreciated that, if there is not existing storage file (or existing storage file cannot or be not suitable for being divided at least two different regions) before step 110, should first create data storage file, and described data storage file at least can be divided into two different regions (first area and second area) hereinafter.
In step 120, the described data that will store of primary importance storage in the first area of data storage file, and the second place in the second area of described data storage file is stored described mark and the additional information relevant with the described data that will store.
In embodiment preferably, shown in can thinking, primary importance and the second place are respectively the ad-hoc locations in first area and second area, such as continuing storage or store at the afterbody of file after the content of last stored.
If there is no this agreement, tackle primary importance and the second place is determined.This definite can realization by the field that remains in advance hereof record position information, and after each file storage success, the information of this field is being upgraded, use for storage next time.
Preferably, in step 120, the first described data that will store of the primary importance storage in the first area of data storage file.If described data storage success, stores described mark and described additional information in the described second place.
Described additional information comprises about the information of described primary importance with about the information of the length of the described data that will store.Preferably, described information can be position off-set value.
Fig. 2 shows the process flow diagram of method for reading data proposed by the invention.Described method for reading data is the method for reading data for the data storage file of storing according to the date storage method shown in Fig. 1 and embodiment thereof.
In step 210, in the described second area of described data storage file, look for the mark that the data that will read are assigned.
Preferably, use and search described mark for the binary search of ordered sequence.
In step 220, determine the primary importance of the described data that will read in the described first area of described data storage file according to the described additional information of storing together with described mark.
In step 230, read the described data that will read.
In Fig. 3, illustrate according to the schematic diagram of the file layout of the data storage file of one embodiment of the present invention.As shown in Figure 1, this file layout is made up of file header region, data block index region and data area.Wherein, described file header area stores file essential information, the field comprising has the data block number n(that current file has write to account for 4 bytes), current file allows the maximum data piece number m(writing to account for 4 bytes) and the length k(of data block ID account for 2 bytes).Data block index region is made up of m index information, and each index information comprises data block ID, data block skew (accounting for 4 bytes) and the data block length (accounting for 4 bytes) in data area.Described data area is stored the data block at every turn writing successively.
Can find out, described data block index region is corresponding to second area above, and described data area is corresponding to first area above, and described additional information comprises skew and the data block length of data block in data area.
Below with reference to Fig. 3, a kind of embodiment of date storage method of the present invention is elaborated.It is pointed out that hereinafter part steps adds for integrality and the continuity of the embodiment for this particular file format, it is not intended to content of the present invention to limit, but for the ease of those skilled in the art's enforcement and understanding.
First, if data storage file does not exist, create file, writing in files head region information, load document head region information is in internal memory, and initialization computation index information writes skew and data block writes skew (corresponding to primary importance and the second place).Wherein:
Index information writes skew=file header zone length+written data piece number n*(data block ID length+4)
Data block writes skew=tail of file
Then, write skew writing data blocks from data block;
And, from index information write skew successively writing data blocks ID, data block at skew and the data block length (corresponding to additional information) of data area;
In order to determine the offset information (primary importance and the second place) when data storage next time, more new data block writes and is offset and data message writes skew:
Data block after renewal writes skew=old data block and writes skew+data block and write length
Index information after renewal writes skew=old index information and writes skew+data block ID length+4
Finally, the written data piece number n field in updating file head region.
Below with reference to Fig. 3, a kind of embodiment of method for reading data of the present invention is elaborated.
First, open data storage file, the file essential information in file reading head region, comprises that written data piece number m, file can write maximum data piece number n and data block ID length k.If written data piece number m is 0, read end, return to sky;
The id information that then, need to read in the domain lookup of data block index area.Data block index region is made up of m index information, and each index information is isometric, can be by binary search fast finding ID in m index information.
Utilize the detailed process of binary search to be:
Read x1 and x2 the position that index information is middle, i.e. the ID of x index information;
X=(x1+x2)/2, the initial value that x1 initial value is 1, x2 is m;
If the ID value of x index information equates with the ID that will search, search end, return to index information, carry out next step;
If the ID value of x index information is less than the ID that will search, x1=x+1, continues to calculate centre position, and searches;
If the ID value of x index information is greater than the ID that will search, x2=x – 1, continues to calculate centre position, and searches;
If x1>x2, searches end, return to sky.
Finally, according to the index information finding, read block is at skew and the data block length of data area, and reads the data block of designated length in skew place.
Corresponding with above-mentioned date storage method, the invention allows for a kind of data storage device, above appropriate section in the explaining of date storage method of the present invention is also applicable to this date storage method.Specifically, this device comprises: mark assignment module, is used to the data assigned identification that will store; Data memory module, stores described mark and the additional information relevant with the described data that will store for the described data that will store of primary importance storage of the first area at data storage file and the second place in the second area of described data storage file.
Preferably, described data storage device also comprises: document creation module, for creating described data storage file, wherein said data storage file at least comprises described first area and described second area.
Preferably, described data storage device also comprises: position determination module, and for determining described primary importance and the described second place.
Preferably, described data memory module is configured to the described data that will store of primary importance storage in the first area of data storage file; If described data storage success, stores described mark and described additional information in the described second place.
Preferably, described mark assignment module is configured to all data and assigns the mark with equal length.
Preferably, to be configured to according to the sequencing of data storage be that their assign the mark that increases progressively of size to described mark assignment module.
Preferably, described additional information comprises about the information of described primary importance with about the information of the length of the described data that will store.
Equally, corresponding with above-mentioned method for reading data, the invention allows for a kind of data fetch device, above appropriate section in the explaining of method for reading data of the present invention is also applicable to this method for reading data.Specifically, this device comprises: mark is searched module, the mark of assigning for look for the data that will read at the described second area of described data storage file; Position determination module, for determining the primary importance of the described data that will read in the described first area of described data storage file according to the described additional information of storing together with described mark; And data read module, for reading the described data that will read.
Preferably, described mark is searched module and is configured to use ordered sequence binary search to search described mark.
Claims (19)
1. a date storage method, the method comprises:
A) the data assigned identification for storing;
B) the described data that will store of the primary importance storage in the first area of data storage file, and the second place in the second area of described data storage file is stored described mark and the additional information relevant with the described data that will store.
2. method according to claim 1, the method also comprises:
C) before step a), create described data storage file, wherein said data storage file at least comprises described first area and described second area.
3. method according to claim 1, the method also comprises:
D), before step b), determine described primary importance and the described second place.
4. method according to claim 1, the method also comprises:
E), after step b), determine primary importance and the second place for data storage next time.
5. according to the method described in any one in claim 1-4, wherein step b) comprises:
B1) the described data that will store of the primary importance storage in the first area of data storage file;
B2) if step b1) in described data storage success, store described mark and described additional information in the described second place.
6. according to the method described in any one in claim 1-4, for all data are assigned the mark with equal length.
7. according to the method described in any one in claim 1-4, according to the sequencing of data storage, for they assign the big or small mark increasing progressively.
8. according to the method described in any one in claim 1-4, wherein said additional information comprises about the information of described primary importance with about the information of the length of the described data that will store.
9. a data storage device, this device comprises:
Mark assignment module, is used to the data assigned identification that will store;
Data memory module, stores described mark and the additional information relevant with the described data that will store for the described data that will store of primary importance storage of the first area at data storage file and the second place in the second area of described data storage file.
10. device according to claim 9, this device also comprises:
Document creation module, for creating described data storage file, wherein said data storage file at least comprises described first area and described second area.
11. devices according to claim 9, this device also comprises:
Position determination module, for determining described primary importance and the described second place.
12. according to the device described in any one in claim 9-11, and wherein said data memory module is configured to:
The described data that will store of primary importance storage in the first area of data storage file;
If described data storage success, stores described mark and described additional information in the described second place.
13. according to the device described in any one in claim 9-11, and wherein said mark assignment module is configured to all data and assigns the mark with equal length.
14. according to the device described in any one in claim 9-11, and it is the mark that they assign size to increase progressively that wherein said mark assignment module is configured to according to the sequencing of data storage.
15. according to the device described in any one in claim 9-11, and wherein said additional information comprises about the information of described primary importance with about the information of the length of the described data that will store.
The method for reading data of 16. 1 kinds of data storage files of storing for the date storage method described in any one according in claim 1-8, the method comprises:
In the described second area of described data storage file, look for the mark that the data that will read are assigned;
Determine the primary importance of the described data that will read in the described first area of described data storage file according to the described additional information of storing together with described mark; And
Read the described data that will read.
17. methods according to claim 16, are wherein used ordered sequence binary search to search described mark.
The data fetch device of 18. 1 kinds of data storage files of storing for the date storage method described in any one according in claim 1-8, this device comprises:
Mark is searched module, the mark of assigning for look for the data that will read at the described second area of described data storage file;
Position determination module, for determining the primary importance of the described data that will read in the described first area of described data storage file according to the described additional information of storing together with described mark; And
Data read module, for reading the described data that will read.
19. methods according to claim 18, wherein mark is searched module and is configured to use ordered sequence binary search to search described mark.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310741482.1A CN103793468A (en) | 2013-12-27 | 2013-12-27 | Data storage method and device and data reading method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310741482.1A CN103793468A (en) | 2013-12-27 | 2013-12-27 | Data storage method and device and data reading method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN103793468A true CN103793468A (en) | 2014-05-14 |
Family
ID=50669134
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310741482.1A Pending CN103793468A (en) | 2013-12-27 | 2013-12-27 | Data storage method and device and data reading method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103793468A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104731534A (en) * | 2015-04-22 | 2015-06-24 | 浪潮电子信息产业股份有限公司 | Method and device for managing video data |
| CN105159608A (en) * | 2015-08-28 | 2015-12-16 | 北京奇艺世纪科技有限公司 | Method and device for reading and writing P2P data |
| CN106372002A (en) * | 2016-08-31 | 2017-02-01 | 成都科来软件有限公司 | Data storage method and data reading and restoring method |
| CN106919633A (en) * | 2016-06-08 | 2017-07-04 | 阿里巴巴集团控股有限公司 | The read method and device of data |
| CN110134660A (en) * | 2019-05-22 | 2019-08-16 | 苏州浪潮智能科技有限公司 | A kind of data access method, device, equipment and computer readable storage medium |
| CN112511612A (en) * | 2020-11-19 | 2021-03-16 | 中国联合网络通信集团有限公司 | Cloud storage data storage method, device, system, equipment and storage medium |
-
2013
- 2013-12-27 CN CN201310741482.1A patent/CN103793468A/en active Pending
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104731534A (en) * | 2015-04-22 | 2015-06-24 | 浪潮电子信息产业股份有限公司 | Method and device for managing video data |
| CN105159608A (en) * | 2015-08-28 | 2015-12-16 | 北京奇艺世纪科技有限公司 | Method and device for reading and writing P2P data |
| CN106919633A (en) * | 2016-06-08 | 2017-07-04 | 阿里巴巴集团控股有限公司 | The read method and device of data |
| CN106372002A (en) * | 2016-08-31 | 2017-02-01 | 成都科来软件有限公司 | Data storage method and data reading and restoring method |
| CN106372002B (en) * | 2016-08-31 | 2019-03-19 | 成都科来软件有限公司 | A kind of date storage method and read restoring method |
| CN110134660A (en) * | 2019-05-22 | 2019-08-16 | 苏州浪潮智能科技有限公司 | A kind of data access method, device, equipment and computer readable storage medium |
| CN112511612A (en) * | 2020-11-19 | 2021-03-16 | 中国联合网络通信集团有限公司 | Cloud storage data storage method, device, system, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10642515B2 (en) | Data storage method, electronic device, and computer non-volatile storage medium | |
| CN103793468A (en) | Data storage method and device and data reading method and device | |
| CN108255958B (en) | Data query method, device and storage medium | |
| CN109542979B (en) | Fast synchronization and simple data storage mode of block chain system | |
| CN110188108B (en) | Data storage method, device, system, computer equipment and storage medium | |
| US7937522B2 (en) | Method for flash memory data management | |
| EP3260993A1 (en) | File operation method and device | |
| US20180329921A1 (en) | Method and apparatus for storing time series data | |
| CN103885887B (en) | User data storage method, read method and system | |
| US11210228B2 (en) | Method, device and computer program product for cache management | |
| CN103699585A (en) | Methods, devices and systems for file metadata storage and file recovery | |
| CN104239234B (en) | The management of a kind of local cache efficiently and reading/writing method | |
| CN105138282A (en) | Storage space recycling method and storage system | |
| CN109240607B (en) | File reading method and device | |
| CN104331453A (en) | Distributed file system and constructing method thereof | |
| CN104102586A (en) | Address mapping processing method and address mapping processing device | |
| CN103810246A (en) | Index building method and device and index query method and device | |
| CN105740334A (en) | System and method for asynchronous and batched file creation in file system | |
| CN113721862A (en) | Data processing method and device | |
| CN104050057A (en) | Historical sensed data duplicate removal fragment eliminating method and system | |
| CN107122140A (en) | A kind of file intelligent storage method based on metadata information | |
| CN111158606B (en) | Storage method, storage device, computer equipment and storage medium | |
| CN116662327A (en) | Data fusion cleaning method for database | |
| CN105279166B (en) | File management method and system | |
| EP3296878B1 (en) | Electronic device and page merging method therefor |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C53 | Correction of patent for invention or patent application | ||
| CB02 | Change of applicant information |
Address after: 100085 Haidian District East Road, No. three, China control building, floor, floor, 1 Applicant after: BEIJING TOPSEC SOFTWARE CO., LTD. Applicant after: Beijing Topsec Network Safety Technology Co., Ltd. Applicant after: BEIJING TOPSEC TECHNOLOGY CO., LTD. Address before: 100085 Haidian District East Road, No. three, China control building, floor, floor, 1 Applicant before: BEIJING TOPSEC SOFTWARE CO., LTD. Applicant before: Beijing Topsec Network Safety Technology Co., Ltd. Applicant before: Beijing heaven melts letter Science Technologies Co., Ltd. |
|
| C53 | Correction of patent for invention or patent application | ||
| CB02 | Change of applicant information |
Address after: 100085 Haidian District East Road, No. three, China control building, floor, floor, 1 Applicant after: BEIJING TOPSEC SOFTWARE CO., LTD. Applicant after: Beijing Topsec Network Safety Technology Co., Ltd. Applicant after: Beijing heaven melts letter Science Technologies Co., Ltd. Address before: 100085 Haidian District East Road, No. three, China control building, floor, floor, 1 Applicant before: BEIJING TOPSEC SOFTWARE CO., LTD. Applicant before: Beijing Topsec Network Safety Technology Co., Ltd. Applicant before: BEIJING TOPSEC TECHNOLOGY CO., LTD. |
|
| CB02 | Change of applicant information |
Address after: 100085, room 306, north 3, building seven, 3 East Road, Haidian District, Beijing Applicant after: BEIJING TOPSEC SOFTWARE CO., LTD. Applicant after: Beijing Topsec Network Safety Technology Co., Ltd. Applicant after: BEIJING TOPSEC TECHNOLOGY CO., LTD. Address before: 100085 Haidian District East Road, No. three, China control building, floor, floor, 1 Applicant before: BEIJING TOPSEC SOFTWARE CO., LTD. Applicant before: Beijing Topsec Network Safety Technology Co., Ltd. Applicant before: Beijing heaven melts letter Science Technologies Co., Ltd. |
|
| COR | Change of bibliographic data | ||
| CB02 | Change of applicant information |
Address after: 100085 Beijing East Road, No. 1, building No. 306, building on the north side of the floor, room 3, room 3 Applicant after: BEIJING TOPSEC SOFTWARE CO., LTD. Applicant after: Beijing Topsec Network Safety Technology Co., Ltd. Applicant after: BEIJING TOPSEC TECHNOLOGY CO., LTD. Address before: 100085, room 306, north 3, building seven, 3 East Road, Haidian District, Beijing Applicant before: BEIJING TOPSEC SOFTWARE CO., LTD. Applicant before: Beijing Topsec Network Safety Technology Co., Ltd. Applicant before: BEIJING TOPSEC TECHNOLOGY CO., LTD. |
|
| COR | Change of bibliographic data | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20140514 |
|
| RJ01 | Rejection of invention patent application after publication |