CN102567434B - Data block processing method - Google Patents
Data block processing method Download PDFInfo
- Publication number
- CN102567434B CN102567434B CN 201110036318 CN201110036318A CN102567434B CN 102567434 B CN102567434 B CN 102567434B CN 201110036318 CN201110036318 CN 201110036318 CN 201110036318 A CN201110036318 A CN 201110036318A CN 102567434 B CN102567434 B CN 102567434B
- Authority
- CN
- China
- Prior art keywords
- data block
- state
- data
- index
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a data block processing method, which includes: creating a data block in internal storage; distributing unique data block identification information of the data block, marking the data block state of the data block as the internal storage stage and updating a second index; receiving to-be-written data items by the data block and updating a first index according to the key words and the storage position of the data block; marking the state of the data block as the written-in state, updating the second index, and writing the data block into a solid-state storage unit; marking the state of the data block as the solid-state storage state after the data block is written into the solid-state storage unit, releasing internal buffering space occupied by the data block and updating the second index to record file descriptors of the solid-state storage unit. By the data block processing method, the data block comprising a plurality of data items is managed by matching of different state marks and internal indexes, high-performance read-write operation can be compatible, and the requirements of high-performance random inquiry and updating of data are met.
Description
Technical field
The present invention relates to technical field of data processing, particularly a kind of data block processing method.
Background technology
High speed development along with the internet, people's information source has obtained abundant greatly, the acquisition of information mode also changes thereupon, when bringing opportunity to the mankind, this also brings challenges, become in Web information under the prerequisite of how much radix growths, how can carry out fast and accurately data search, to search requirement, be one of direction of technical field of data processing research.
In data search, search engine spider is more and more used, and spider is an auto-programming of search engine, and its effect is the webpage on the crawl internet, set up index data base, make the user can in search engine, search the webpage of related web site.
In specific implementation process, spider will grasp a large amount of web site urls every day, all needed to obtain the information such as the IP address of website to be grasped and robots before crawl, these information can not be real-time from interconnected online enquiries, and can only be by inner domain name server (DNS) inquiry.
But because the data volume of storing is in continuous increase among the DNS, so inquiry velocity also can be thereupon slack-off, can not satisfy the demand of fast query.And, when the data among the DNS are upgraded, also can increase the workload of DNS, this has also affected the speed of inquiry.Equally, also can run into similar problem in web database and other key word-data values (Key-value) database in real time.
How can better data being inquired about and upgrade, satisfy high performance read-write service, is one of direction of technical field of data processing research.
Summary of the invention
Technical matters to be solved by this invention provides a kind of data block processing method, to support the high-performance read-write operation, satisfy to the high-performance random challenge of data with upgrade demand.
The present invention is that the technical scheme that the technical solution problem adopts provides a kind of data block processing method, comprise: a. creates data block in internal memory, store the first index and the second index in the wherein said internal memory, described the first index is used for the mapping relations of the positional information of recording key and corresponding data item, positional information comprises the data block identifying information, data item offset information and data item length information, described the second index is finished data block identifying information from described positional information to the mapping of data block information, and described data block information comprises described data block identifying information, described data block state, data block length information and internal memory pointer; B. be that described data block is distributed unique data block identifying information, and be internal storage state with the data block status indication of described data block, upgrade described the second index; C. utilize described data block to receive data item to be written, upgrade described the first index according to key word and the memory location of described data item; D. described data block write full after, be write state with the data block status indication of described data block, upgrade described the second index, and described data block be written in the solid-state memory; E. after described data block is written to described solid-state memory, be the solid-state storage state with the data block status indication of described data block, discharge the memory buffer space that described data block takies, and upgrade described the second index, described data block information further comprises the solid-state memory filec descriptor.
The preferred embodiment one of according to the present invention in described step c, upgrades described the second index, and described data block information further comprises effective data item number, the effective bitmap of data item, internal storage data piece capacity.
The preferred embodiment one of according to the present invention when described data item is carried out tag delete, carries out mark to the corresponding position in the effective bitmap of described data item in described the second index.
The preferred embodiment one of according to the present invention in described steps d, judges by the comparative result of described internal storage data piece capacity and described data block length information whether described data block is write full.
The preferred embodiment one of according to the present invention, in described step c, when the valid data item number in the described data block is lower than threshold value, be the reconstruction state with the data block status indication of described data block, subsequently the valid data item in the described data block is dumped in the new data block that is labeled as internal storage state, and after unloading is finished, be the deletion state with the data block status indication of described data block.
The preferred embodiment one of according to the present invention, in described steps d, when the valid data item number in the described data block is lower than threshold value, be the reconstruction state with the data block status indication of described data block, subsequently the valid data item in the described data block is dumped in the new data block that is labeled as internal storage state, and after unloading is finished, be the deletion state with the data block status indication of described data block.
The preferred embodiment one of according to the present invention, in described step e, when the valid data item number in the described data block is lower than threshold value, be reading state with the data block status indication of described data block, described data block is read in the described internal memory, and be the reconstruction state reading after finishing with the data block status indication of described data block, subsequently the valid data item in the described data block is dumped in the new data block that is labeled as internal storage state, and after unloading is finished, be the deletion state with the data block status indication of described data block.
The preferred embodiment one of according to the present invention, described disposal route further comprises: the f. judge mark is whether the described data block of deletion state is cited, and does not then delete described data block if be cited.
The preferred embodiment one of according to the present invention in described step e, is written to described data block in the described solid-state memory with controlled speed.
As can be seen from the above technical solutions, data block processing method provided by the invention utilizes different status indications to cooperate the internal memory index that the data block that comprises a plurality of data item is managed, can support the high-performance read-write operation, satisfy the high-performance random challenge of data and the demand of renewal.
Description of drawings
Fig. 1 is the schematic flow sheet of the database control method in the embodiment of the invention;
Fig. 2 is the storage medium of the database in the embodiment of the invention and the synoptic diagram of memory contents;
Fig. 3 is the first index in the embodiment of the invention and the data structure synoptic diagram of data block;
Fig. 4 is the data structure synoptic diagram of the second index in the embodiment of the invention;
Fig. 5 is the structural representation of the Hash container in the embodiment of the invention;
Fig. 6 is the synoptic diagram of the data block life cycle management process in the embodiment of the invention;
Fig. 7 is the schematic flow sheet of the database read extract operation in the embodiment of the invention;
Fig. 8 is the schematic flow sheet that the database in the embodiment of the invention writes operation;
Fig. 9 is the idiographic flow synoptic diagram of the step S809 among Fig. 8;
Figure 10 is the schematic flow sheet that the database of the embodiment of the invention is derived operation;
Figure 11 is the structural representation of the database control device in the embodiment of the invention.
Embodiment
The present invention is described in detail below in conjunction with drawings and Examples.
See also Fig. 1, Fig. 1 is the schematic flow sheet of the database control method in the embodiment of the invention.In the present embodiment, the control method of database mainly comprises following step:
Step S101 receives operation information.
Step S102 is according to the index information of operation information at the index inquiry corresponding data piece that is arranged in internal memory.
Step S103 carries out corresponding operating according to operation information and index information to the corresponding data piece.
In the present invention, operation information can comprise the key word of concrete operations instruction and pending data item or pending data item.The concrete operations instruction can comprise read operation instruction, write operation instruction, derive operational order etc., and the specific operation process that various operation informations are corresponding will be described below.
See also Fig. 2, Fig. 2 is the storage medium of the database in the embodiment of the invention and the synoptic diagram of memory contents.In the present embodiment, the storage medium of database comprises internal memory and solid-state memory.Memory contents comprises index and data block.Wherein, data block is arranged in internal memory and solid-state memory according to the different conditions selectivity.Specifically, when data block is arranged in internal memory, can be described as the internal storage data piece, when data block is arranged in solid-state memory, can be described as the solid-state memory data block.Wherein, index is arranged in the internal memory, and index comprises the first index and the second index.
See also Fig. 3-4, Fig. 3 is the first index in the embodiment of the invention and the data structure synoptic diagram of data block.Fig. 4 is the data structure synoptic diagram of the second index in the embodiment of the invention.
See also Fig. 3, in the present embodiment, the first index is used for the mapping relations of the positional information of recording key and corresponding data item.Positional information mainly comprises following information: data block identifying information, data item offset information and data item length information.
Wherein, the data block identifying information is used for the ID of the data block under the record corresponding data item, and the data item offset information is used for record corresponding data item in the side-play amount of data block, and the data item length information is used for the length of record corresponding data item.
See also Fig. 3, the second index is finished data block identifying information from positional information to the mapping of data block information.See also Fig. 4, in the present embodiment, data block information mainly comprises: data block identifying information, data block length information, data item sum, valid data item number, the effective bitmap of data item, data block state, internal storage data piece capacity, internal memory pointer, solid-state memory filec descriptor and reference count.
Wherein, the data block identifying information is used for the ID of recording data blocks, and in the present embodiment, each data block is distributed a unique ID.Data block length information is used for the record data block size.The data item sum is used for total number of the data item in the recording data blocks.Valid data item number is used for the number of the valid data item in the recording data blocks, and namely the data item sum deducts the number of the data item that is labeled deletion.The effective bitmap of data item is used for the effective status of record data items, and wherein, each (bit) represents a data item, if put 1 expression effectively, sets to 0 expression and is labeled deletion.The data block state is used for the state of recording data blocks, is mainly used in judging that the corresponding data piece is arranged in internal memory or is arranged in solid-state memory, hereinafter will describe various data block states in detail.Internal storage data piece capacity is used for the data item size that record internal storage data piece has been stored, is mainly used in cooperating to judge with data block length information whether the internal storage data piece is full.The internal memory pointer is used for recording data blocks in the memory location of internal memory, and when data block was not arranged in internal memory, it was invalid then should to be worth.The solid-state memory filec descriptor is used for recording data blocks in the memory location of solid-state memory, and when data block was not arranged in solid-state memory, it was invalid then should to be worth.Reference count is used for the state of quoting of recording data blocks, is used for the life cycle of management data block.
In above-mentioned information, data block identifying information, data block length information, data item sum, valid data item number, the effective bitmap of data item are the core data item of data block, are called the metamessage of data block.
Please continue to consult Fig. 3, a plurality of data item of storage store following content: data item sequence number, key length, data value length, key word and data value in each data item in data block.
Wherein, the data item sequence number is the sequence number of data item in data block, is used for searching the effective bitmap of data item.Key length is used for the length of recording key.Data value length is used for the length of record data value.Key word is the binary string of expression key word.Data value is the binary string of presentation data value.
In the present invention, the first index and the second index can be by the various algorithms realizations in this area, for example hash algorithms.
One embodiment of the present invention provides a kind of Hash container of indirect addressing, has saved the space of index, has improved the service efficiency of internal memory.The below will take the first index as example, be described in detail.
See also Fig. 5, in the present embodiment, the first index comprises a Hash bucket table, and this Hash bucket table comprises a plurality of Hash buckets.Storage one Hash node pointer in each Hash bucket.In the present embodiment, the Hash node pointer be predetermined bite (for example, 4 bytes), wherein front pre-determined bit (for example, front 9) for the Hash node data piece that identifies Hash node place, rear pre-determined bit (for example, rear 23) is used for sign Hash node in the skew of Hash node data piece inside.
Specifically, the identifying information of all Hash node data pieces is recorded in the list of identification information, and can inquire according to the front pre-determined bit of Hash node pointer the identifying information of correspondence from the correspondence position of list of identification information.In the present embodiment, the maximum quantity of the identifying information that can store of list of identification information is 2
9=512.
In addition, the Hash node is stored in the corresponding Hash node data piece, each Hash node (for example takies predetermined bite, 20 bytes), (for example comprise respectively key word (for example, 8 bytes), data block identifying information, 2 bytes), the data item length information (for example, 2 bytes), data item offset information (for example, 4 bytes) and next Hash node pointer (for example, 4 bytes).
In the use procedure of above-mentioned Hash container, at first the key word that receives is carried out Hash operation and from Hash bucket table, determine corresponding Hash bucket, and from the Hash bucket, obtain the Hash node pointer.Subsequently, utilize the identifying information of front pre-determined bit Hash node data piece of definite correspondence from list of identification information of Hash node pointer, and from Hash node data piece corresponding to identifying information, obtain corresponding Hash node according to the rear pre-determined bit of Hash node pointer as side-play amount, and then obtain the data item positional information relevant with this key word, for example data block identifying information, data item offset information and data item length information.
The maximum quantity of the Hash node that can store in each Hash node data piece in the present embodiment, is 2
23=8388608, so the maximum amount of data that each Hash container can be supported is 512 * 8,388,608,=42 hundred million, well satisfied the demand of name server (Domain Name Server, DNS).
In addition, in this Hash container, the idle linked list maintenance of idle Hash node.When data were deleted, corresponding Hash node also can be recovered.Idle Hash node conspires to create one by next the Hash node pointer in the Hash node and reclaims chained list.When receiving new data, the preferential pointer that reclaims in the chained list that uses.Therefore, the Hash node data in the Hash node data piece is always compact, in website quantity is in 300,000,000 the situation, to save as 300M * 20bytes=6Gbytes in taking.
See also Fig. 6, Fig. 6 is the synoptic diagram of the data block life cycle management process in the embodiment of the invention.
In the present embodiment, at first, in internal memory, create data block.After creating data block, be that this data block is distributed unique data block identifying information (ID), and be " internal memory " with the data block status indication of this data block.Subsequently, upgrade the second index, to record the data block information of this data block, such as data block identifying information, data block length information, data block state and internal memory pointer etc.Wherein, database only has at most a data block to be in " internal memory " state at any one time.
The data block that is labeled as " internal memory " can receive data item to be written, and this data item is appended to the data block end.Subsequently, upgrade the second index, to record effective data item number, the effective bitmap of data item, internal storage data piece capacity etc.Simultaneously, the first index is upgraded in key word and memory location according to data item to be written, the mapping relations of this key word of record and data block identifying information, data item offset information and data item length information in the first index can be according to this keyword query to corresponding data item so that follow-up.In case behind the data item writing data blocks, just cannot change, only allow this data item is read and tag delete.When this data item is carried out tag delete, in the second index, mark is carried out in the corresponding position in the effective bitmap of data item of the data block information of this data block, for example with correspondence position 0.
Behind the continuous writing data blocks of data item, can judge whether this data block is write full by the comparative result of internal storage data piece capacity and data block length information.If it is full that data block is write, be " writing " with the data block status indication of this data block, and upgrade the second index, total with record data items.Subsequently, this data block is written in the solid-state memory.Preferably, in ablation process, the data item of this data block in the internal memory (for example, 5MB/S) is written in the solid-state memory, has effectively prevented from reading performance is caused excessive impact with controlled speed.Simultaneously, in internal memory, again create the data block status indication for the new data block of " internal memory ", to receive the follow-up data item that writes.
After the data item of this data block of " writing " state all is written to solid-state memory, is " solid-state storage " with the data block status indication of this data block, and discharges the memory buffer space that this data block originally took.Subsequently, upgrade the second index, to record the data block information of this data block, solid-state memory filec descriptor for example.
If the cavity that is in the data block of " internal memory " state and " writing " state is too many, that is to say that the number of the valid data item in the data block is lower than threshold value, then the data block status indication with this data block is " reconstruction ".Subsequently, the interior valid data item of data block that will be labeled as " reconstruction " dumps in the new data block that is labeled as " internal memory ", to realize removing and the reconstruction to the invalid data items in the data block of " reconstruction ".After reconstruction was finished, the data block status indication that will be labeled as the data block of " reconstruction " was " deletion ".At this moment, judged whether that according to the reference count of this data block other threads quote this data block, if there are not other threads to quote this data block, then deleted this data block.If there are other threads to quote this data block, then keep this data block, until after other threads are used to complete, remove by reference operation and discharge this reference count, again this data block is deleted.
If the cavity that is in the data block of " solid-state storage " state is too many, then the data block status indication with this data block is " reading ", and this data block is read in the internal memory.Read finish after, be " reconstruction " with the data block status indication of this data block, and the valid data item of this data block dumped in the data block that is labeled as " internal memory ", to realize removing and the reconstruction to the interior invalid data items of the data block of " reconstructions ".After reconstruction was finished, the data block status indication that will be labeled as the data block of " reconstruction " was " deletion ".At this moment, judged whether that according to the reference count of this data block other threads quote this data block, if there are not other threads to quote this data block, then deleted this data block.
Specifically, if the data block state is " internal memory ", " writing " and " reconstruction ", represent that then this data block is arranged in internal memory.If the data block state is " solid-state storage " and " reading ", represent that then this data block is arranged in solid-state memory.See through aforesaid operations, effectively management data block life cycle.
Below in conjunction with specific embodiment, various operating process of the present invention is described.
See also Fig. 7, Fig. 7 is the schematic flow sheet of the database read extract operation in the embodiment of the invention.
In step S701, receive the key word of read operation instruction and data item to be read.
In step S702, utilize this key word from positional information corresponding to the first search index.If do not inquire, then carry out step S708; If inquire, then carry out step S703.
In the present embodiment, positional information comprises data block identifying information, data item offset information and data item length information.Concrete query script is described in detail hereinbefore, does not repeat them here.
In step S703, utilize data block identifying information in this positional information from data block information corresponding to the second search index.In the present embodiment, data block information comprises data block state, internal memory pointer and solid-state memory filec descriptor etc.
In step S704, be arranged in internal memory or be positioned at solid-state memory according to data block condition judgement corresponding data piece, if the corresponding data piece is arranged in internal memory, then carry out step S705, if the corresponding data piece is arranged in solid-state memory, then carry out step S706.
As described above, if the data block state is " internal memory ", " writing " and " reconstruction ", represent that then this data block is arranged in internal memory.If the data block state is " solid-state storage " and " reading ", represent that then this data block is arranged in solid-state memory.
In step S705, in internal memory, read the corresponding data item according to internal memory pointer, data item offset information and data item length information.
In step S706, in solid-state memory, read the corresponding data item according to solid-state memory filec descriptor, data item offset information and data item length information.
In step S707, judge whether the key word of corresponding data item is consistent with the key word of data item to be read, if inconsistent, then carry out step S708; If consistent, then carry out step S709.
In step S708, judge that data item to be read does not exist.
In step S709, judge and to read successfully, and with the data value of the corresponding data item data value as data item to be read.
See also Fig. 8, Fig. 8 is the schematic flow sheet that the database of the embodiment of the invention writes operation.
In step S801, receive key word and the data value of write operation instruction and data item to be written.
In step S802, utilize this key word from positional information corresponding to the first search index.If inquire, then carry out step S803; If do not inquire, then carry out step S809.
In the present embodiment, positional information comprises data block identifying information, data item offset information and data item length information.Concrete query script is described in detail hereinbefore, does not repeat them here.
In step S803, utilize data block identifying information in this positional information from data block information corresponding to the second search index.In the present embodiment, data block information comprises data block state, internal memory pointer and solid-state memory filec descriptor etc.
In step S804, be arranged in internal memory according to data block condition judgement corresponding data piece and still be arranged in solid-state memory, if the corresponding data piece is arranged in internal memory, then carry out step S805, if the corresponding data piece is arranged in solid-state memory, then carry out step S806.
As described above, if the data block state is " internal memory ", " writing " and " reconstruction ", represent that then this data block is arranged in internal memory.If the data block state is " solid-state storage " and " reading ", represent that then this data block is arranged in solid-state memory.
In step S805, in internal memory, read the corresponding data item according to internal memory pointer, data item offset information and data item length information.
In step S806, in solid-state memory, read the corresponding data item according to solid-state memory filec descriptor, data item offset information and data item length information;
In step S807, judge whether the data value of corresponding data item is consistent with the data value of data item to be written:, then write successfully if consistent; If inconsistent, then carry out step S808;
In step S808, under the corresponding data item, in the effective bitmap of the data item of data block this corresponding data item sign is deleted.Specifically, in the effective bitmap of data item with correspondence position 0.
In step S809, data item to be written is write the data block that is used for receiving data item to be written that is arranged in internal memory, namely be in the data block of " internal memory " state.Subsequently, upgrade the first index and the second index to record above-mentioned ablation process.
See also Fig. 9, Fig. 9 is the idiographic flow synoptic diagram of the step S809 among Fig. 8.
In step S901, judge whether the data block of being in the internal memory " internal memory " state has been write completely, expire if write, then carry out step S902, expire if write, then carry out step S903.Specifically, can determine whether this data block is write full by the comparative result between data block length information and the internal storage data piece capacity.
In step S902, directly data item to be written is written to the end of data block, and upgrades the first index and the second index.
In step S903, be " writing " state with the data block status indication of this data block.Subsequently, enter step S904 and step S905.
In step S904, this data block is write solid-state memory, further upgrade simultaneously the first index and the second index.Write finish after, change the data block state of this data block into " solid-state storage " state, in internal memory, discharge simultaneously this data block.
In step S905, create the new data block that is labeled as " internal memory " state, and data item to be written is written to the end of new data block, and upgrade the first index and the second index.
By the way, step S904 and step S905 can walk abreast and carry out, and have realized that thus read-write separates.In addition, the data item that is in the data block of " writing " state writes solid-state memory with controlled speed, and for example controlled speed is 5MB/S.The present invention is by arranging controlled speed, the impact that when effectively having prevented data writing reading performance has been caused.
See also Figure 10, Figure 10 is the schematic flow sheet that the database of the embodiment of the invention is derived operation.
In step S1001, receiving derives operational order.
In step S1002, internal memory and solid-state memory are locked, wherein, under locking state, forbid internal memory and solid-state memory are made amendment.
In step S1003, from the second index of internal memory, copy metamessage corresponding to data to be exported piece.In the present embodiment, the data to be exported piece can be the data block of all data blocks or predetermined quantity.As indicated above, metamessage comprises data block identifying information, data block length information, data item sum, valid data item number and the effective bitmap of data item.
In step S1004, the invoking marks operation is carried out in the reference count of data to be exported piece, to avoid the data to be exported piece deleted.For example, reference count is added one or add particular step size.
In step S1005, internal memory and solid-state memory are carried out release, wherein, under released state, allow internal memory and solid-state memory are made amendment.Because the doubling time of metamessage is very short, the follow-up derivation of data block can be carried out on the backstage, has avoided affecting reading and write operation of data block.
In step S1006, metamessage is write meta-information file.
In step S1007, read the data to be exported piece according to metamessage from internal memory or solid-state memory, and generate the derivation index according to the key word of data to be exported piece.
In step S1008, will derive index and write the derivation index file.
In step S1009, with data to be exported piece data writing block file.In the present embodiment, meta-information file, derivation index file and data block file can create when receiving the derivation operational order, also can be in other arbitrarily suitably constantly establishments.
In step S1010, the releasing operation is quoted in the reference count of data to be exported piece.For example, reference count is subtracted one or subtract particular step size.At this moment, if data block is in " deletion " state, and do not have other threads to quote this data block, then delete this data block.
See also Figure 11, Figure 11 is the structural representation of the database control device of the embodiment of the invention.This control device comprises operation information receiver module 1101, index information enquiry module 1102 and data block processing module 1103.
Wherein, operation information receiver module 1101 is used for receiving aforesaid operations information.
Index information enquiry module 1102 is used for according to the index information of aforesaid operations information at the index inquiry corresponding data piece that is positioned at internal memory.In the present embodiment, the corresponding data piece comprises a plurality of data item, and each data item comprises key word and data value, and corresponding data piece alternative is arranged in internal memory and solid-state memory.
Data block processing module 1103 is used for according to aforesaid operations information and index information the corresponding data piece being carried out corresponding operating.
In specific implementation process, index comprises the first above-mentioned index and the second index.The positional information that index information enquiry module 1102 is corresponding with key word according to the first search index, positional information comprise data block identifying information, data item offset information and data item length information.Index information enquiry module 1102 is the data block information corresponding with the data block identifying information according to the second search index further, and data block information comprises data block state, internal memory pointer and solid-state memory filec descriptor etc.
See also above description about the detailed operation process of database control device, repeat no more herein.
As can be seen from the above technical solutions, the control method of database of the present invention and device are stored a plurality of data item with the data block form, and according to different conditions data block selection is stored in internal memory and the solid-state storage, further cooperate the internal memory index, can support the high-performance read-write operation, satisfied to the high-performance random challenge of data with upgrade demand.
In the above-described embodiments, only the present invention has been carried out exemplary description, but those skilled in the art can carry out various modifications to the present invention after reading present patent application in the situation that does not break away from the spirit and scope of the present invention.
Claims (9)
1. a data block processing method is characterized in that, the treating method comprises:
A. in internal memory, create data block, store the first index and the second index in the wherein said internal memory, described the first index is used for the mapping relations of the positional information of recording key and corresponding data item, positional information comprises data block identifying information, data item offset information and data item length information, described the second index is finished data block identifying information from described positional information to the mapping of data block information, and described data block information comprises described data block identifying information, described data block state, data block length information and internal memory pointer;
B. be that described data block is distributed unique data block identifying information, and be internal storage state with the data block status indication of described data block, upgrade described the second index;
C. utilize described data block to receive data item to be written, upgrade described the first index according to key word and the memory location of described data item;
D. described data block write full after, be write state with the data block status indication of described data block, upgrade described the second index, and described data block be written in the solid-state memory;
E. after described data block is written to described solid-state memory, be the solid-state storage state with the data block status indication of described data block, discharge the memory buffer space that described data block takies, and upgrade described the second index, described data block information further comprises the solid-state memory filec descriptor.
2. disposal route as claimed in claim 1 is characterized in that, in described step c, upgrades described the second index, and described data block information further comprises effective data item number, the effective bitmap of data item, internal storage data piece capacity.
3. disposal route as claimed in claim 2 is characterized in that, when described data item is carried out tag delete, in described the second index mark is carried out in the corresponding position in the effective bitmap of described data item.
4. disposal route as claimed in claim 2 is characterized in that, in described steps d, judges by the comparative result of described internal storage data piece capacity and described data block length information whether described data block is write full.
5. disposal route as claimed in claim 2, it is characterized in that, in described step c, when the valid data item number in the described data block is lower than threshold value, be the reconstruction state with the data block status indication of described data block, subsequently the valid data item in the described data block is dumped in the new data block that is labeled as internal storage state, and after unloading is finished, be the deletion state with the data block status indication of described data block.
6. disposal route as claimed in claim 2, it is characterized in that, in described steps d, when the valid data item number in the described data block is lower than threshold value, be the reconstruction state with the data block status indication of described data block, subsequently the valid data item in the described data block is dumped in the new data block that is labeled as internal storage state, and after unloading is finished, be the deletion state with the data block status indication of described data block.
7. disposal route as claimed in claim 2, it is characterized in that, in described step e, when the valid data item number in the described data block is lower than threshold value, be reading state with the data block status indication of described data block, described data block is read in the described internal memory, and be the reconstruction state reading after finishing with the data block status indication of described data block, subsequently the valid data item in the described data block is dumped in the new data block that is labeled as internal storage state, and after unloading is finished, be the deletion state with the data block status indication of described data block.
8. such as the described disposal route of claim 5-7 any one, it is characterized in that described disposal route further comprises:
F. judge mark is whether the described data block of deletion state is cited, and does not then delete described data block if be cited.
9. disposal route as claimed in claim 1 is characterized in that, in described step e, described data block is written in the described solid-state memory with controlled speed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201110036318 CN102567434B (en) | 2010-12-31 | 2010-12-31 | Data block processing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201110036318 CN102567434B (en) | 2010-12-31 | 2010-12-31 | Data block processing method |
Related Parent Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN 201010619673 Division CN102567415B (en) | 2010-12-31 | 2010-12-31 | Control method and device of database |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102567434A CN102567434A (en) | 2012-07-11 |
| CN102567434B true CN102567434B (en) | 2013-04-10 |
Family
ID=46412863
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN 201110036318 Active CN102567434B (en) | 2010-12-31 | 2010-12-31 | Data block processing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102567434B (en) |
Families Citing this family (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103403709B (en) * | 2012-11-15 | 2016-10-05 | 华为技术有限公司 | A kind of methods, devices and systems of reading and writing data |
| CN103853766B (en) * | 2012-12-03 | 2017-04-05 | 中国科学院计算技术研究所 | A kind of on-line processing method and system towards stream data |
| CN102968498B (en) * | 2012-12-05 | 2016-08-10 | 华为技术有限公司 | Data processing method and device |
| WO2014178843A1 (en) * | 2013-04-30 | 2014-11-06 | Hewlett-Packard Development Company, L.P. | Database table column annotation |
| CN103810246B (en) * | 2013-12-27 | 2017-10-13 | 北京天融信软件有限公司 | A kind of index creation method and apparatus and search index method and apparatus |
| CN105426408B (en) * | 2015-11-02 | 2019-03-08 | 北京锐安科技有限公司 | A kind of data processing method and device of more indexes |
| CN106453674B (en) * | 2016-09-09 | 2019-05-17 | 中国互联网络信息中心 | A kind of DNS multistage domain name inquiry method |
| CN106484770B (en) * | 2016-09-09 | 2019-08-06 | 中国互联网络信息中心 | A kind of processing method of DNS incremental area data file |
| CN106528650B (en) * | 2016-10-14 | 2019-06-21 | 努比亚技术有限公司 | A kind of resource query method and terminal |
| CN108628969B (en) * | 2018-04-24 | 2022-05-06 | 咪咕文化科技有限公司 | Spatial keyword indexing method and platform and storage medium |
| CN108829740B (en) * | 2018-05-23 | 2021-11-02 | 摇了购（武汉）电子商务有限公司 | Data storage method and device |
| CN109885558B (en) * | 2019-01-21 | 2024-06-04 | 南京国通智能科技有限公司 | Data state configuration method, system and storage medium |
| CN110059087B (en) * | 2019-03-26 | 2023-02-28 | 创新先进技术有限公司 | Data attribute identification method, device and equipment in block chain type account book |
| CN111124313A (en) * | 2019-12-24 | 2020-05-08 | 宁波三星医疗电气股份有限公司 | Data reading and writing method and device for power acquisition terminal and electronic equipment |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1270363A (en) * | 1999-04-09 | 2000-10-18 | 英业达股份有限公司 | Database management method |
| CN1538324A (en) * | 2003-04-16 | 2004-10-20 | ��Ϊ��������˾ | Method of raising officiency of data processing |
| CN1645365A (en) * | 2004-12-20 | 2005-07-27 | 浪潮电子信息产业股份有限公司 | Method for decreasing DBMS field storing redundance |
| CN1908952A (en) * | 2005-07-21 | 2007-02-07 | 富士施乐株式会社 | Information processing server, information processing system, information processing method and storage medium storing information processing program |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH02287875A (en) * | 1989-04-28 | 1990-11-27 | Fuji Xerox Co Ltd | Filing system in computer system |
| JPH08235040A (en) * | 1995-02-27 | 1996-09-13 | Hitachi Ltd | Data file management system |
-
2010
- 2010-12-31 CN CN 201110036318 patent/CN102567434B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1270363A (en) * | 1999-04-09 | 2000-10-18 | 英业达股份有限公司 | Database management method |
| CN1538324A (en) * | 2003-04-16 | 2004-10-20 | ��Ϊ��������˾ | Method of raising officiency of data processing |
| CN1645365A (en) * | 2004-12-20 | 2005-07-27 | 浪潮电子信息产业股份有限公司 | Method for decreasing DBMS field storing redundance |
| CN1908952A (en) * | 2005-07-21 | 2007-02-07 | 富士施乐株式会社 | Information processing server, information processing system, information processing method and storage medium storing information processing program |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102567434A (en) | 2012-07-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102567434B (en) | Data block processing method | |
| CN102541968B (en) | Indexing method | |
| CN102567415B (en) | Control method and device of database | |
| US9858303B2 (en) | In-memory latch-free index structure | |
| JP5996088B2 (en) | Cryptographic hash database | |
| CN103080910B (en) | Storage system | |
| US9043334B2 (en) | Method and system for accessing files on a storage system | |
| US9830324B2 (en) | Content based organization of file systems | |
| KR20210057835A (en) | Compressed Key-Value Store Tree Data Block Leak | |
| CN105224546B (en) | Data storage and query method and equipment | |
| CN110119425A (en) | Solid state drive, distributed data-storage system and the method using key assignments storage | |
| CN104424219B (en) | A kind of management method and device of data file | |
| KR20160046729A (en) | Key-value storage engine and efficient key collision handling method thereof | |
| CN103164490B (en) | A kind of efficient storage implementation method of not fixed-length data and device | |
| CN103186617B (en) | A kind of method and apparatus storing data | |
| JP2005122702A5 (en) | ||
| CN102292720A (en) | Method and apparatus for managing data objects of a data storage system | |
| CN112131140B (en) | SSD-based key value separation storage method supporting efficient storage space management | |
| CN101419571A (en) | Method for storing configuration parameter in NOR FLASH based on Hash arithmetic | |
| CN104572920A (en) | Data arrangement method and data arrangement device | |
| CN109407985B (en) | Data management method and related device | |
| CN103617097A (en) | File recovery method and file recovery device | |
| CN103279366A (en) | Solid state disk and method for rapidly start-up based on solid state disk after power failure | |
| CN109408416B (en) | Address mapping table entry page management method and device | |
| CN110633261A (en) | Picture storage method, picture query method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant |