CN104407991A - Data storage method and device - Google Patents
Data storage method and device Download PDFInfo
- Publication number
- CN104407991A CN104407991A CN201410756207.1A CN201410756207A CN104407991A CN 104407991 A CN104407991 A CN 104407991A CN 201410756207 A CN201410756207 A CN 201410756207A CN 104407991 A CN104407991 A CN 104407991A
- Authority
- CN
- China
- Prior art keywords
- data
- hash table
- layer
- value
- attribute
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application provides a data storage method and a device thereof. The method comprises the following steps: storing a plurality of captured data into a first cache based on the sequence of capturing data, and generating a first nested hash table based on the locations of the plurality of data in the first cache, the first hash table is a nested multi-layer hash table, the key contents of each layer of hash table are corresponding to the attribute types of the captured data one-to-one, and the values of the innermost layer of hash table are address value of each data in the first cache; obtaining the address values in the innermost layer of hash table based on a preset traversal sequence, thus sequentially storing the data corresponding to each address value in the first cache to a hard drive based on the obtaining sequence of each address value. By comparing with the manner of the prior art that each data is stored only based on the sequence of capturing, the efficiency of reading data stored by the data storage method provided by the embodiment of the invention can be increased.
Description
Technical field
The application relates to technical field of data storage, especially a kind of utilize data attribute to carry out data storage method and device.
Background technology
Network data analysis field, usually needs the data in crawl network and stores, so that follow-up reading is analyzed.At present, the storage mode of network data is, by the data that grab according to crawl sequencing, stores successively in a hard disk.But this kind of storage mode is also unfavorable for the reading of data, more particularly, reads the data efficiency utilizing this kind of mode to store lower.
Summary of the invention
In view of this, the invention provides a kind of date storage method and device, in order to solve the technical matters that the data utilizing available data storage means to store are inconvenient to read.For realizing described object, technical scheme provided by the invention is as follows:
A kind of date storage method, comprising:
According to the sequencing captured, many data grabbed are stored to the first buffer memory; Wherein, data described in each bar have the attribute of N number of identical type;
According to the position of described many data in described first buffer memory, generate the first nested Hash table; Wherein, described first nested Hash table forms by N layer first Hash table is nested, the key of described first Hash table of each layer is followed successively by the attribute of described N number of identical type, and the 1st layer of value to described first Hash table of N-1 layer is followed successively by first Hash table of lower one deck, the value of n-th layer first Hash table is the address value of data described in each bar in described first buffer memory;
According to default traversal order, obtain the address value in described n-th layer first Hash table successively;
According to the order obtaining address value described in each, successively the data that in described first buffer memory, described in each, address value is corresponding are stored to hard disk.
Alternatively, the data that in described first buffer memory, described in each, address value is corresponding are stored to hard disk and comprise by the described order according to obtaining address value described in each successively:
According to the order obtaining address value described in each, successively the data that in described first buffer memory, described in each, address value is corresponding are stored in the second buffer memory;
Data in described second buffer memory are stored to hard disk successively.
Alternatively, also comprise:
Generate and store the second nested Hash table; Wherein, described second nested Hash table forms by N layer second Hash table is nested, the key of every layer of described second Hash table is tlv triple, and the 1st layer of value to described second Hash table of N-1 layer is followed successively by second Hash table of lower one deck, the value of n-th layer second Hash table is the address value of data in described hard disk described in each bar;
First element of the described tlv triple of each layer is followed successively by the attribute of described N number of identical type, second element of every layer of described tlv triple is the start address of data in described hard disk that the attribute kit of respective layer corresponding types contains, and the 3rd element of every layer of described tlv triple is the data length of data in described hard disk that the attribute kit of respective layer corresponding types contains.
Alternatively, also comprise:
According to described second nested Hash table, from described hard disk, read the data of storage.
Alternatively, described according to described second nested Hash table, the data reading storage from described hard disk comprise:
When receiving data read command, extract the attribute type in described data read command and property value;
In described second nested Hash table, determine the target tlv triple corresponding with described attribute type and property value;
According to the second element and the 3rd element of described target tlv triple, from described hard disk, read the data of storage.
Present invention also provides a kind of data storage device, comprising:
Many the data grabbed, for according to the sequencing captured, are stored to the first buffer memory by the first buffer memory unit; Wherein, data described in each bar have the attribute of N number of identical type;
First Hash table generation unit, for according to the position of described many data in described first buffer memory, generates the first nested Hash table; Wherein, described first nested Hash table forms by N layer first Hash table is nested, the key of described first Hash table of each layer is followed successively by the attribute of described N number of identical type, and the 1st layer of value to described first Hash table of N-1 layer is followed successively by first Hash table of lower one deck, the value of n-th layer first Hash table is the address value of data described in each bar in described first buffer memory;
Buffer address value acquiring unit, for according to presetting traversal order, obtains the address value in described n-th layer first Hash table successively;
The data that in described first buffer memory, described in each, address value is corresponding, for according to the order obtaining address value described in each, are stored to hard disk by hard disk data storage unit successively.
Alternatively, described hard disk data storage unit comprises:
The data that in described first buffer memory, described in each, address value is corresponding, for according to the order obtaining address value described in each, are stored in the second buffer memory by the second buffer memory subelement successively;
Hard disc data storing sub-units, for being stored to hard disk successively by the data in described second buffer memory.
Alternatively, also comprise:
Second Hash table generation unit, for generating and storing the second nested Hash table; Wherein, described second nested Hash table forms by N layer second Hash table is nested, the key of every layer of described second Hash table is tlv triple, and the 1st layer of value to described second Hash table of N-1 layer is followed successively by second Hash table of lower one deck, the value of n-th layer second Hash table is the address value of data in described hard disk described in each bar;
First element of the described tlv triple of each layer is followed successively by the attribute of described N number of identical type, second element of every layer of described tlv triple is the start address of data in described hard disk that the attribute kit of respective layer corresponding types contains, and the 3rd element of every layer of described tlv triple is the data length of data in described hard disk that the attribute kit of respective layer corresponding types contains.
Alternatively, also comprise:
Hard disc data reading unit, for according to described second nested Hash table, reads the data of storage from described hard disk.
Alternatively, described hard disc data reading unit comprises:
Reading command receives subelement, for when receiving data read command, extracts the attribute type in described data read command and property value;
Target tlv triple determination subelement, in described second nested Hash table, determines the target tlv triple corresponding with described attribute type and property value;
Hard disc data reads subelement, for according to the second element of described target tlv triple and the 3rd element, reads the data of storage from described hard disk.
From above technical scheme, the invention provides a kind of date storage method and device, the method comprises: according to the sequencing capturing data, many the data grabbed are stored in the first buffer memory, and according to the position of many data in the first buffer memory, generate the first nested Hash table, this first Hash table is nested multilayer Hash table, the key content of each layer Hash table and the attribute type one_to_one corresponding of the data grabbed, and the value of innermost layer Hash table is the address value of pieces of data in the first buffer memory, the address value in innermost layer Hash table is obtained according to the traversal order preset, and then according to the acquisition of each address value order, successively data corresponding for each address value in the first buffer memory are stored to hard disk.Only compare according to the mode of the sequencing storage pieces of data captured in prior art, read the data of the data storage method storage utilizing the embodiment of the present invention to provide, the reading efficiency of data can be improved.
Accompanying drawing explanation
In order to be illustrated more clearly in the embodiment of the present invention or technical scheme of the prior art, be briefly described to the accompanying drawing used required in embodiment or description of the prior art below, apparently, accompanying drawing in the following describes is only embodiments of the invention, for those of ordinary skill in the art, under the prerequisite not paying creative work, other accompanying drawing can also be obtained according to the accompanying drawing provided.
The process flow diagram of the date storage method that Fig. 1 provides for the embodiment of the present invention;
The process flow diagram of the reading storage data method that Fig. 2 provides for the embodiment of the present invention;
The structural drawing of the data storage device that Fig. 3 provides for the embodiment of the present invention;
The structural drawing of the device of the reading storage data that Fig. 4 provides for the embodiment of the present invention.
Embodiment
Below in conjunction with the accompanying drawing in the embodiment of the present invention, be clearly and completely described the technical scheme in the embodiment of the present invention, obviously, described embodiment is only the present invention's part embodiment, instead of whole embodiments.Based on the embodiment in the present invention, those of ordinary skill in the art, not making the every other embodiment obtained under creative work prerequisite, belong to the scope of protection of the invention.
In network data analysis field, need the data grabbed from network to store.In existing data storage method, by the data that the grab sequencing according to the time of crawl, store in a hard disk successively, but this kind of storage mode can not meet reading demand well.
Specifically, usually need according to certain attributive analysis network data, if network data utilizes storage mode of the prior art to store, during reading, successively all data must all be read one time, to judge whether the property value of each data meets attribute conditions, thus filter out the network data of needs.This kind of reading manner needs to read relatively large data from hard disk, carry out a large amount of Data Matching computings, add the load that hard disk I/O reads and writes pressure and CPU, the reading of data is analyzed consuming time longer, that is, the reading efficiency of network data is lower, and this shortcoming is more obvious when store network data amount is larger.
In order to improve the reading efficiency of network data, each embodiment below the present invention provides data storage method, invention thought is by temporary in the buffer according to time order and function order for the data grabbed, then record has the data position in the buffer of identical type attribute, and then utilize the content of record, by the data in buffer memory according to being stored in hard disk after attribute type sequence, be about to the data Coutinuous store with identical type attribute, like this, also can read data continuously according to attribute conditions when reading, thus improve reading efficiency.
Refer to Fig. 1, it illustrates the flow process of the date storage method that one embodiment of the invention provides, specifically comprise following flow process:
Step S101: according to the sequencing captured, many data grabbed are stored to the first buffer memory; Wherein, data described in each bar have the attribute of N number of identical type.
In the present embodiment, when needs are analyzed network data, can gripping tool be utilized, from network, capture data, and according to grabbing time order and function order, data are stored in buffer memory.Wherein, can preset time period in, whenever gripping tool grabs a network data, by the buffer memory that this network data is put into.Or gripping tool grabs many network datas in preset time period, then, according to crawl order, these many data are put in buffer memory.
It should be noted that, network data has attribute, and many data of crawl have the attribute of identical type, and same alike result type comprises N number of, N be greater than 1 integer.To grab the packet in TCP session, each packet comprises " source IP address ", " source port ", " target ip address " and " target port ", and this four attribute is the attribute of identical type.Certainly, be only an example, concrete attribute type is relevant to the data scene of actual monitoring herein, and the present invention is also not specifically limited.
Step S102: according to the position of described many data in described first buffer memory, generates the first nested Hash table; Wherein, described first nested Hash table forms by N layer first Hash table is nested, the key of described first Hash table of each layer is followed successively by the attribute of described N number of identical type, and the 1st layer of value to described first Hash table of N-1 layer is followed successively by first Hash table of lower one deck, the value of n-th layer first Hash table is the address value of data described in each bar in described first buffer memory.
In the buffer memory that the data grabbed are put into by previous step, the data in this buffer memory are according to the arrangement of crawl time data.As described above, in order to data be resequenced, need record to have the data position in the buffer of identical type attribute, wherein a kind of recording mode is Hash table.Hash table utilizes key-value pair to store data, and namely can store several Hash records in Hash table, each Hash record has a key and a value, and both have corresponding relation.The Hash table generated in this step is nested multilayer Hash table, and nested form is, the 1st layer of value to N-1 layer Hash table is from inside to outside followed successively by the Hash table of lower one deck.
It should be noted that, the number of the attribute type that the number of plies of nested Hash table is identical with network data is equal, the N value namely in step S101.The key of the Hash table in every layer is the attribute of a type, and like this, the nested Hash table of generation utilizes the data of the identical type attribute of the key record in every layer of Hash table.Hash record number in every layer of Hash table can be one, and also can be multiple, in every layer, the key of each Hash record be the attribute of identical type, but property value is not identical.It should be noted that, the value of innermost layer and n-th layer Hash table is data addresses in the buffer.
Wherein, the process generating the first nested Hash table is: data store in the buffer, and that is, every bar data are all to the address in a buffer memory.Because every bar data all have the attribute of N type, according to the property value of the every type attribute of data self, the Hash record that in layer key for searching content is identical with the property value of data, until find the Hash record in n-th layer Hash table, these data address is in the buffer stored in the value of this Hash record found, so repeatedly can generates nested Hash table.It should be noted that, the form of nested Hash table (i.e. the first nested Hash table) and the detailed process generating nested Hash table vide infra.
Step S103: according to default traversal order, obtains the address value in described n-th layer first Hash table successively.
Wherein, the first nested Hash table that step S102 generates comprises multilayer Hash table, and Hash table has default traversal order, similar with array traversal order, travels through successively according to the content of key.Particularly, from the Hash table of ground floor, when the Hash in the Hash table of ground floor is recorded as multiple, then according to preset order, from first Hash record.When the value of this Hash record is lower one deck Hash table, then enter in lower one deck Hash table, judge from first record of lower one deck Hash table, when the value of the Hash record in this lower one deck Hash table is lower one deck Hash table, enter this lower one deck Hash table, judge the value of the Hash record in this lower one deck Hash table, repeat above-mentioned steps, until the value of Hash record in certain layer of Hash table is data addresses in the buffer, now, obtain the address in this buffer memory, get the value of first Hash record in n-th layer like this.In like manner, the value of other Hash records in n-th layer is got.
After traversal terminates, the value got is each data address values in the buffer, and the acquisition of address value order gets according to the attribute type classification of data.
Step S104: according to the order obtaining address value described in each, successively the data that in described first buffer memory, described in each, address value is corresponding are stored to hard disk.
Data in each address value be stored to successively in hard disk, the order of storage is for getting the order of each address value.Like this, by the original data according to capturing sequencing storage, according to attribute type Coutinuous store, scattered data condense together according to attribute type by this kind of storage mode, thus can facilitate when reading and find the data with same alike result type rapidly, data reading performance using redundancy is higher.
Certainly, step S103 and step S104 alternately performs, and namely whenever step S103 gets address value, just according to this address value, searches the data that this address value is corresponding in the buffer, and then data are stored to hard disk by execution step S104.
From above technical scheme, the date storage method that the embodiment of the present invention provides, according to the sequencing capturing data, many the data grabbed are stored in the first buffer memory, and according to the position of many data in the first buffer memory, generate the first nested Hash table, this first Hash table is nested multilayer Hash table, the key content of each layer Hash table and the attribute type one_to_one corresponding of the data grabbed, and the value of innermost layer Hash table is the address value of pieces of data in the first buffer memory, the address value in innermost layer Hash table is obtained according to the traversal order preset, and then according to the acquisition of each address value order, successively data corresponding for each address value in the first buffer memory are stored to hard disk.Only compare according to the mode of the sequencing storage pieces of data captured in prior art, read the data of the data storage method storage utilizing the embodiment of the present invention to provide, the reading efficiency of data can be improved.
The attribute type of data can be time, physical address, data direction, IP session data bag, TCP session data bag etc., then reading demand can be all packets that in certain time period, certain physical address sends, then the attribute type stored in the first nested Hash table is time, physical address and data direction respectively; Or reading demand is all packets of certain physics address accept in certain time period, then the attribute type stored in the first nested Hash table is time, physical address and data direction respectively; Or reading demand is all packets of certain IP session in certain time period, then reading demand is time and IP session data bag; Or reading demand is all packets of certain TCP session in certain time period, then reading demand is time and TCP session data bag.Certainly, above several properties type and reading demand be only example, concrete situation can be different because of the data grabbed, and the embodiment of the present invention is also not specifically limited.
Below the form of the nested Hash table of first in above-described embodiment is described.
Wherein, the form of the first nested Hash table can with reference to following table 1.Such as, the data grabbed comprise the attribute of three identical types, are respectively attribute A, attribute B and attribute C, this Hash table as shown in table 1, this Hash table comprises three layers, wherein, from outside to inside, the key of ground floor Hash table belongs to attribute A, and the key of each Hash record is respectively a
1, a
2a
n, the value of each Hash record is second layer Hash table, and the key table of second layer Hash table shows attribute B, and the key of each Hash record is respectively b
1, b
2b
n, the value of each Hash record is third layer Hash table, and the key table of third layer Hash table shows attribute C, and the key of each Hash record is respectively c
1, c
2c
n, the value of each Hash record is the buffer address of data.
Table 1
From innermost layer, the property value of the pieces of data that data acquisition 1 comprises is respectively a
1, b
1and c
1, the property value of the pieces of data that data acquisition 2 comprises is respectively a
1, b
1and c
2, the property value of the pieces of data that data acquisition n comprises is respectively a
1, b
1and c
n.
From outermost layer, data are polymerized according to attribute A, and namely all data are classified according to the value of attribute A, and the data that the value of attribute A is different belong to different Hash records respectively.From the second layer, on the basis of classification of ground floor, data are polymerized according to attribute B, and the data that namely value of attribute B is different belong to different Hash records respectively.The like, until the Hash table of interior one deck.It should be noted that, each Hash record in every one deck Hash table has storage order, and this order can be capture time order and function order.
More specifically, with following table 2, table 3 and table 4, each layer Hash table can be described.Wherein:
From outermost layer, the buffer address of data carries out poly-sequencing by merging according to the value of attribute A.
Table 2
| " attribute A " value is a 1The buffer address of data |
| " attribute A " value is a 2The buffer address of data |
| … |
| " attribute A " value is a nThe buffer address of data |
Enter lower one deck to see, the buffer address of data carries out poly-sequencing by merging according to the value of attribute B.
Table 3
| " attribute B " value is b 1The buffer address of data |
| " attribute B " value is b 2The buffer address of data |
| … |
| " attribute B " value is b nThe buffer address of data |
Enter lower one deck to see, the buffer address of data carries out poly-sequencing by merging according to the value of attribute C.
Table 4
| " attribute C " value is c 1The buffer address of data |
| " attribute C " value is c 2The buffer address of data |
| … |
| " attribute C " value is c nThe buffer address of data |
As can be seen from the nested Hash table shown in above-mentioned table 1, nested Hash table utilizes the buffer address of the data of the key difference recording different types attribute of every layer of Hash table, and when the property value of the attribute of multiple data same type is identical, the buffer address value of these multiple data can be stored in same Hash record.
Below the detailed process generating the first nested Hash table shown in above-mentioned table 1 is described.
Suppose that the data grabbed have the attribute of three types, be respectively attribute A, attribute B and attribute C.The data grabbed are respectively (d
1, d
2, d
3.d
n), pieces of data is kept in buffer memory, just can get every bar data address in the buffer.
Set up a Hash table HA, this Hash table HA is as outermost Hash table, and the key of this Hash table HA, for storing the value of the attribute A of the data grabbed, is worth the Hash table HB for storing lower one deck; The key of this Hash table HB, for storing the value of the attribute B of the data grabbed, is worth the Hash table HC for storing lower one deck; The key of this Hash table HC, for storing the value of the attribute C of the data grabbed, is worth for storing data address in the buffer.
The first step, obtains the Article 1 data d grabbed
1, the value of the attribute A of these Article 1 data is Ad, and in this Hash table HA, key for searching is the Hash record of Ad, if find, then carries out subsequent step, otherwise, in this Hash table HA, add key be Ad and value is the record of empty Hash table.
Second step, suppose that previous step finds in Hash table HA (or add) being recorded as rhb, the value of record rhb is also a Hash table HB.This Article 1 data d
1the value of attribute B be Bd, in Hash table HB, key for searching is the Hash record of Bd, if find, then carries out subsequent step, otherwise, in this Hash table HB, add a key be Bd and value is the record of empty Hash table.
3rd step, suppose that previous step finds in Hash table HB (or add) being recorded as rhc, the value of record rhc is also a Hash table HC.This Article 1 data d
1the value of attribute C be Cd, in Hash table HC, key for searching is the Hash record of Cd, if find, then carries out subsequent step, otherwise, in this Hash table HC, add a key be Cd and the record that closes for null data set of value.
4th step, suppose that previous step finds in Hash table HC (or add) being recorded as rds, then the value of rds is data acquisition, and that comprise in this data acquisition is d
1address in the buffer, so far, completes the storing process of the buffer address of data.Following continuation obtains next the data d grabbed
2, in like manner, by data d
2buffer address store according to above-mentioned steps, namely generate nested Hash table after the buffer address storage of the data grabbed being completed.
Step S104 in above-described embodiment is according to the order obtaining address value described in each, and the specific implementation successively data that in described first buffer memory, described in each, address value is corresponding being stored to hard disk can comprise:
According to the order obtaining address value described in each, successively the data that in described first buffer memory, described in each, address value is corresponding are stored in the second buffer memory; Data in described second buffer memory are stored to hard disk successively.
Particularly, the data in the first buffer memory be stored in the second buffer memory, namely the order of storage obtains the order of address value, is also the traversal order of nested Hash table.Because nested Hash table is according to attribute type classification memory address value, then according to this traversal order, data corresponding for each address value are stored into after in the second buffer memory, the data meeting Coutinuous store of same type attribute is belonged in second buffer memory, data in this second buffer memory are stored to hard disk successively, thus the data stored in hard disk are also according to attribute type Coutinuous store.
In this kind of implementation, utilize the second buffer memory that the data in the first buffer memory are stored to hard disk, when avoiding directly the data in the first buffer memory being stored to hard disk, repeatedly write the hard disk I/O interface pressure that hard disk causes.
The conveniently reading of data, can generate and store the second Hash table, and this second Hash table is for data corresponding to the attribute that records every type start address in a hard disk and data length.
Particularly, described second nested Hash table forms by N layer second Hash table is nested, the key of every layer of described second Hash table is tlv triple, and the 1st layer of value to described second Hash table of N-1 layer is followed successively by second Hash table of lower one deck, the value of n-th layer second Hash table is the address value of data in described hard disk described in each bar; First element of the described tlv triple of each layer is followed successively by the attribute of described N number of identical type, second element of every layer of described tlv triple is the start address of data in described hard disk that the attribute kit of respective layer corresponding types contains, and the 3rd element of every layer of described tlv triple is the data length of data in described hard disk that the attribute kit of respective layer corresponding types contains.
Wherein, the start address that the second nested Hash table utilizes the tlv triple record attribute value of key, data store and data length.The concrete form of this table can be joined and is shown in Table 5.
Table 5
As can be seen from this table, the property value of attribute A is respectively (a1, a2 ... an), the property value of attribute B (b1, b2 respectively ... bn), the property value of attribute C is respectively (c1, c2 ... cn); Property value is that the start address of the continuous data of a1 is addr_a1, data length is len_a1, and property value is that the start address of the continuous data of a1 and b1 is addr_b1, data length is len_b1, by that analogy, and other data known.
This nested Hash table is utilized to carry out digital independent, described in specific as follows.
According to the attribute type of the data grabbed, utilize each embodiment above-mentioned that the described data grabbed are stored to hard disk, to facilitate digital independent, namely when receiving data read command, according to the second nested Hash table generated, from hard disk, read the data of storage.Particularly, as shown in Figure 2, this reading process comprises:
Step S201: when receiving data read command, extracts the attribute type in described data read command and property value.
Wherein, data read command comprises attribute type and property value.Such as, attribute type is time period and four-tuple and source IP address, source port, target ip address and target port.
Step S202: in described second nested Hash table, determine the target tlv triple corresponding with described attribute type and property value.
For the second nested Hash table shown in above-mentioned table 5, search attribute type in this table and be the attribute type that extracts and property value is the tlv triple of property value extracted, suppose that attribute type is A and property value is a1, then the tlv triple found is (a1, addr_a1, len_a1); And for example attribute type is A and B, and property value is respectively a1 and b2, then the tlv triple found is (b2, addr_b2, len_b2).
Step S203: according to the second element and the 3rd element of described target tlv triple, read the data of storage from described hard disk.
Namely with the start address of the second element of target tlv triple for starting point, from hard disk, read the continuous data of data length of the 3rd element representation.Namely this section of continuous data be the data needing to read.
Such as, the attribute type of the data grabbed comprises time and TCP session data bag, further, the attribute type of TCP session data bag is four-tuple, the data type so stored comprises time and four-tuple, when storage mode data provided according to above-described embodiment is stored into after in hard disk, data are reset according to attribute type, namely there is the data Coutinuous store of same time and/or identical four-tuple, and in the second nested Hash table, record start address and the data length of the same time of Coutinuous store and/or the data of identical four-tuple, like this when reading the data of certain four-tuple in certain time period, start address and the data length of this segment data only need be found in the second nested Hash table, the continuous data of this segment data length is read from this start address.
Compared with existing data reading mode, the data reading mode that the embodiment of the present invention provides, do not need all data all to read one time with garbled data, digital independent amount is little thus alleviate the access hole burden of hard disk, and improves data reading performance using redundancy.
Be described the data storage device that the embodiment of the present invention provides below, it should be noted that, the introduction of data storage device hereafter can refer to date storage method provided above, does not repeat herein.
With reference to Fig. 3, it illustrates the structure of the data storage device that the embodiment of the present invention provides, specifically comprise: the first buffer memory unit 101, first Hash table generation unit 102, buffer address value acquiring unit 103 and hard disk data storage unit 104.Wherein:
Many the data grabbed, for according to the sequencing captured, are stored to the first buffer memory by the first buffer memory unit 101; Wherein, data described in each bar have the attribute of N number of identical type;
First Hash table generation unit 102, for according to the position of described many data in described first buffer memory, generates the first nested Hash table; Wherein, described first nested Hash table forms by N layer first Hash table is nested, the key of described first Hash table of each layer is followed successively by the attribute of described N number of identical type, and the 1st layer of value to described first Hash table of N-1 layer is followed successively by first Hash table of lower one deck, the value of n-th layer first Hash table is the address value of data described in each bar in described first buffer memory;
Buffer address value acquiring unit 103, for according to presetting traversal order, obtains the address value in described n-th layer first Hash table successively;
The data that in described first buffer memory, described in each, address value is corresponding, for according to the order obtaining address value described in each, are stored to hard disk by hard disk data storage unit 104 successively.
From above technical scheme, the data storage device that the embodiment of the present invention provides, first buffer memory unit 101 is according to the sequencing capturing data, many the data grabbed are stored in the first buffer memory, first Hash table generation unit 102 is according to the position of many data in the first buffer memory, generate the first nested Hash table, this first Hash table is nested multilayer Hash table, the key content of each layer Hash table and the attribute type one_to_one corresponding of the data grabbed, and the value of innermost layer Hash table is the address value of pieces of data in the first buffer memory, buffer address value acquiring unit 103 obtains the address value in innermost layer Hash table according to the traversal order preset, and then hard disk data storage unit 104 is according to the acquisition order of each address value, successively data corresponding for each address value in the first buffer memory are stored to hard disk.Only compare according to the mode of the sequencing storage pieces of data captured in prior art, read the data of the data storage device stores utilizing the embodiment of the present invention to provide, the reading efficiency of data can be improved.
Alternatively, hard disk data storage unit 104 comprises:
The data that in described first buffer memory, described in each, address value is corresponding, for according to the order obtaining address value described in each, are stored in the second buffer memory by the second buffer memory subelement successively;
Hard disc data storing sub-units, for being stored to hard disk successively by the data in described second buffer memory.
Alternatively, above-mentioned data storage device, also comprises: the second Hash table generation unit.
Second Hash table generation unit, for generating and storing the second nested Hash table; Wherein, described second nested Hash table forms by N layer second Hash table is nested, the key of every layer of described second Hash table is tlv triple, and the 1st layer of value to described second Hash table of N-1 layer is followed successively by second Hash table of lower one deck, the value of n-th layer second Hash table is the address value of data in described hard disk described in each bar;
First element of the described tlv triple of each layer is followed successively by the attribute of described N number of identical type, second element of every layer of described tlv triple is the start address of data in described hard disk that the attribute kit of respective layer corresponding types contains, and the 3rd element of every layer of described tlv triple is the data length of data in described hard disk that the attribute kit of respective layer corresponding types contains.
Alternatively, above-mentioned data storage device, also comprises: hard disc data reading unit, for according to described second nested Hash table, reads the data of storage from described hard disk.
As shown in Figure 4, hard disc data reading unit 200 comprises: reading command receives subelement 201, target tlv triple determination subelement 202 and hard disc data and reads subelement 203; Wherein:
Reading command receives subelement 201, for when receiving data read command, extracts the attribute type in described data read command and property value;
Target tlv triple determination subelement 202, in described second nested Hash table, determines the target tlv triple corresponding with described attribute type and property value;
Hard disc data reads subelement 203, for according to the second element of described target tlv triple and the 3rd element, reads the data of storage from described hard disk.
It should be noted that, each embodiment in this instructions all adopts the mode of going forward one by one to describe, and what each embodiment stressed is the difference with other embodiments, between each embodiment identical similar part mutually see.
Also it should be noted that, in this article, the such as relational terms of first and second grades and so on is only used for an entity or operation to separate with another entity or operational zone, and not necessarily requires or imply the relation that there is any this reality between these entities or operation or sequentially.And, term " comprises ", " comprising " or its any other variant are intended to contain comprising of nonexcludability, thus make to comprise the process of a series of key element, method, article or equipment and not only comprise those key elements, but also comprise other key elements clearly do not listed, or also comprise by the intrinsic key element of this process, method, article or equipment.When not more restrictions, the key element limited by statement " comprising ... ", and be not precluded within process, method, article or the equipment comprising above-mentioned key element and also there is other identical element.
To the above-mentioned explanation of the disclosed embodiments, professional and technical personnel in the field are realized or uses the present invention.To be apparent for those skilled in the art to the multiple amendment of these embodiments, General Principle as defined herein can without departing from the spirit or scope of the present invention, realize in other embodiments.Therefore, the present invention can not be restricted to these embodiments shown in this article, but will meet the widest scope consistent with principle disclosed herein and features of novelty.
Claims (10)
1. a date storage method, is characterized in that, comprising:
According to the sequencing captured, many data grabbed are stored to the first buffer memory; Wherein, data described in each bar have the attribute of N number of identical type;
According to the position of described many data in described first buffer memory, generate the first nested Hash table; Wherein, described first nested Hash table forms by N layer first Hash table is nested, the key of described first Hash table of each layer is followed successively by the attribute of described N number of identical type, and the 1st layer of value to described first Hash table of N-1 layer is followed successively by first Hash table of lower one deck, the value of n-th layer first Hash table is the address value of data described in each bar in described first buffer memory;
According to default traversal order, obtain the address value in described n-th layer first Hash table successively;
According to the order obtaining address value described in each, successively the data that in described first buffer memory, described in each, address value is corresponding are stored to hard disk.
2. date storage method according to claim 1, is characterized in that, the data that in described first buffer memory, described in each, address value is corresponding are stored to hard disk and comprise by the described order according to obtaining address value described in each successively:
According to the order obtaining address value described in each, successively the data that in described first buffer memory, described in each, address value is corresponding are stored in the second buffer memory;
Data in described second buffer memory are stored to hard disk successively.
3. date storage method according to claim 1, is characterized in that, also comprises:
Generate and store the second nested Hash table; Wherein, described second nested Hash table forms by N layer second Hash table is nested, the key of every layer of described second Hash table is tlv triple, and the 1st layer of value to described second Hash table of N-1 layer is followed successively by second Hash table of lower one deck, the value of n-th layer second Hash table is the address value of data in described hard disk described in each bar;
First element of the described tlv triple of each layer is followed successively by the attribute of described N number of identical type, second element of every layer of described tlv triple is the start address of data in described hard disk that the attribute kit of respective layer corresponding types contains, and the 3rd element of every layer of described tlv triple is the data length of data in described hard disk that the attribute kit of respective layer corresponding types contains.
4. date storage method according to claim 3, is characterized in that, also comprises:
According to described second nested Hash table, from described hard disk, read the data of storage.
5. date storage method according to claim 4, is characterized in that, described according to described second nested Hash table, the data reading storage from described hard disk comprise:
When receiving data read command, extract the attribute type in described data read command and property value;
In described second nested Hash table, determine the target tlv triple corresponding with described attribute type and property value;
According to the second element and the 3rd element of described target tlv triple, from described hard disk, read the data of storage.
6. a data storage device, is characterized in that, comprising:
Many the data grabbed, for according to the sequencing captured, are stored to the first buffer memory by the first buffer memory unit; Wherein, data described in each bar have the attribute of N number of identical type;
First Hash table generation unit, for according to the position of described many data in described first buffer memory, generates the first nested Hash table; Wherein, described first nested Hash table forms by N layer first Hash table is nested, the key of described first Hash table of each layer is followed successively by the attribute of described N number of identical type, and the 1st layer of value to described first Hash table of N-1 layer is followed successively by first Hash table of lower one deck, the value of n-th layer first Hash table is the address value of data described in each bar in described first buffer memory;
Buffer address value acquiring unit, for according to presetting traversal order, obtains the address value in described n-th layer first Hash table successively;
The data that in described first buffer memory, described in each, address value is corresponding, for according to the order obtaining address value described in each, are stored to hard disk by hard disk data storage unit successively.
7. data storage device according to claim 6, is characterized in that, described hard disk data storage unit comprises:
The data that in described first buffer memory, described in each, address value is corresponding, for according to the order obtaining address value described in each, are stored in the second buffer memory by the second buffer memory subelement successively;
Hard disc data storing sub-units, for being stored to hard disk successively by the data in described second buffer memory.
8. data storage device according to claim 6, is characterized in that, also comprises:
Second Hash table generation unit, for generating and storing the second nested Hash table; Wherein, described second nested Hash table forms by N layer second Hash table is nested, the key of every layer of described second Hash table is tlv triple, and the 1st layer of value to described second Hash table of N-1 layer is followed successively by second Hash table of lower one deck, the value of n-th layer second Hash table is the address value of data in described hard disk described in each bar;
First element of the described tlv triple of each layer is followed successively by the attribute of described N number of identical type, second element of every layer of described tlv triple is the start address of data in described hard disk that the attribute kit of respective layer corresponding types contains, and the 3rd element of every layer of described tlv triple is the data length of data in described hard disk that the attribute kit of respective layer corresponding types contains.
9. data storage device according to claim 8, is characterized in that, also comprises:
Hard disc data reading unit, for according to described second nested Hash table, reads the data of storage from described hard disk.
10. data storage device according to claim 9, is characterized in that, described hard disc data reading unit comprises:
Reading command receives subelement, for when receiving data read command, extracts the attribute type in described data read command and property value;
Target tlv triple determination subelement, in described second nested Hash table, determines the target tlv triple corresponding with described attribute type and property value;
Hard disc data reads subelement, for according to the second element of described target tlv triple and the 3rd element, reads the data of storage from described hard disk.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410756207.1A CN104407991A (en) | 2014-12-10 | 2014-12-10 | Data storage method and device |
| CN201510907032.4A CN105335300B (en) | 2014-12-10 | 2015-12-09 | A kind of date storage method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410756207.1A CN104407991A (en) | 2014-12-10 | 2014-12-10 | Data storage method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN104407991A true CN104407991A (en) | 2015-03-11 |
Family
ID=52645624
Family Applications (2)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410756207.1A Pending CN104407991A (en) | 2014-12-10 | 2014-12-10 | Data storage method and device |
| CN201510907032.4A Active CN105335300B (en) | 2014-12-10 | 2015-12-09 | A kind of date storage method and device |
Family Applications After (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510907032.4A Active CN105335300B (en) | 2014-12-10 | 2015-12-09 | A kind of date storage method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (2) | CN104407991A (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106484332A (en) * | 2016-10-12 | 2017-03-08 | 郑州云海信息技术有限公司 | A kind of date storage method and device |
| CN106504183A (en) * | 2015-09-08 | 2017-03-15 | 龙芯中科技术有限公司 | The method and device of vertex attribute storage |
| CN107220105A (en) * | 2017-06-22 | 2017-09-29 | 中国人民解放军国防科学技术大学 | Parallel Discrete Event Simulation system auto-initiation implementation method |
| CN107315801A (en) * | 2017-06-22 | 2017-11-03 | 中国人民解放军国防科学技术大学 | Parallel Discrete Event Simulation system initialization date storage method |
| CN107453948A (en) * | 2017-07-28 | 2017-12-08 | 北京邮电大学 | The storage method and system of a kind of network measurement data |
| CN112181312A (en) * | 2020-10-23 | 2021-01-05 | 北京安石科技有限公司 | Method and system for quickly reading hard disk data |
| CN112465344A (en) * | 2020-11-26 | 2021-03-09 | 中船重工鹏力（南京）智能装备系统有限公司 | Method for initializing simulation system parameters based on workshop automation production line |
| CN112491857A (en) * | 2020-11-20 | 2021-03-12 | 北京人大金仓信息技术股份有限公司 | Method, device and equipment for transmitting set type data |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11468781B2 (en) | 2017-08-11 | 2022-10-11 | The Boeing Company | Simulation of an asset including message playback using nested hash tables |
| CN107632792A (en) * | 2017-10-13 | 2018-01-26 | 郑州云海信息技术有限公司 | The method and apparatus that virtual disk is managed in cloud data system |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7099881B2 (en) * | 2002-12-06 | 2006-08-29 | Stmicroelectronics, Inc. | Method for increasing average storage capacity in a bit-mapped tree-based storage engine by using remappable prefix representations and a run-length encoding scheme that defines multi-length fields to compactly store IP prefixes |
| CN1652112A (en) * | 2005-03-02 | 2005-08-10 | 北京北方烽火科技有限公司 | Implementing method of data dictionary under embedded environment |
| CN101777025A (en) * | 2009-12-25 | 2010-07-14 | 宇龙计算机通信科技(深圳)有限公司 | Method and device for storing multilevel menu content and network communication terminal |
| CN102253988A (en) * | 2011-06-30 | 2011-11-23 | 北京新媒传信科技有限公司 | Method for filtering sensitive words in network text service |
| CN104090942A (en) * | 2014-06-30 | 2014-10-08 | 中国电子科技集团公司第三十二研究所 | Trie search method and device applied to network processor |
-
2014
- 2014-12-10 CN CN201410756207.1A patent/CN104407991A/en active Pending
-
2015
- 2015-12-09 CN CN201510907032.4A patent/CN105335300B/en active Active
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106504183A (en) * | 2015-09-08 | 2017-03-15 | 龙芯中科技术有限公司 | The method and device of vertex attribute storage |
| CN106504183B (en) * | 2015-09-08 | 2019-09-10 | 龙芯中科技术有限公司 | The method and device of vertex attribute storage |
| CN106484332A (en) * | 2016-10-12 | 2017-03-08 | 郑州云海信息技术有限公司 | A kind of date storage method and device |
| CN107220105A (en) * | 2017-06-22 | 2017-09-29 | 中国人民解放军国防科学技术大学 | Parallel Discrete Event Simulation system auto-initiation implementation method |
| CN107315801A (en) * | 2017-06-22 | 2017-11-03 | 中国人民解放军国防科学技术大学 | Parallel Discrete Event Simulation system initialization date storage method |
| CN107315801B (en) * | 2017-06-22 | 2019-12-13 | 中国人民解放军国防科学技术大学 | parallel discrete event simulation system initialization data storage method |
| CN107453948A (en) * | 2017-07-28 | 2017-12-08 | 北京邮电大学 | The storage method and system of a kind of network measurement data |
| CN112181312A (en) * | 2020-10-23 | 2021-01-05 | 北京安石科技有限公司 | Method and system for quickly reading hard disk data |
| CN112491857A (en) * | 2020-11-20 | 2021-03-12 | 北京人大金仓信息技术股份有限公司 | Method, device and equipment for transmitting set type data |
| CN112465344A (en) * | 2020-11-26 | 2021-03-09 | 中船重工鹏力（南京）智能装备系统有限公司 | Method for initializing simulation system parameters based on workshop automation production line |
| CN112465344B (en) * | 2020-11-26 | 2023-10-24 | 中船重工鹏力（南京）智能装备系统有限公司 | Initialization method based on workshop automatic production line simulation system parameters |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105335300B (en) | 2018-10-12 |
| CN105335300A (en) | 2016-02-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104407991A (en) | Data storage method and device | |
| CN103488709B (en) | A kind of index establishing method and system, search method and system | |
| US9405790B2 (en) | System, method and data structure for fast loading, storing and access to huge data sets in real time | |
| CN104050269B (en) | log compression method and device, decompression method and device | |
| JP2012526320A5 (en) | ||
| CN104298736B (en) | Data acquisition system connection method, device and Database Systems | |
| JP2016519810A5 (en) | ||
| CN103812877B (en) | Data compression method based on Bigtable distributed memory system | |
| JP2008269197A5 (en) | ||
| CN102467458B (en) | Method for establishing index of data block | |
| CN106445416A (en) | Data record storage, query and retrieval method and device | |
| CN106484332A (en) | A kind of date storage method and device | |
| CN107480466A (en) | Genomic data storage method and electronic equipment | |
| CN102169491A (en) | Dynamic detection method for multi-data concentrated and repeated records | |
| CN104932011B (en) | A kind of method and device for picking up seismic first breaks | |
| CN104424204B (en) | Indexing Mechanism merging method, searching method, device and equipment | |
| CN107590233B (en) | File management method and device | |
| CN106649436A (en) | Data archiving method and device | |
| CN104239236B (en) | The processing method of bypass conversion buffered missing and bypass conversion buffered | |
| CN103942012B (en) | A kind of method, device and equipment for reading data | |
| CN106547810A (en) | A kind of flow stores the method and system of quick indexing | |
| CN107704208A (en) | A kind of restorative procedure of metadata, device and medium | |
| CN109726203A (en) | A kind of date storage method of reconstruct image | |
| Halboob et al. | An efficient computer forensics selective imaging model | |
| CN107203550A (en) | A kind of data processing method and database server |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C02 | Deemed withdrawal of patent application after publication (patent law 2001) | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20150311 |