CN104572920A - Data arrangement method and data arrangement device - Google Patents
Data arrangement method and data arrangement device Download PDFInfo
- Publication number
- CN104572920A CN104572920A CN201410836030.6A CN201410836030A CN104572920A CN 104572920 A CN104572920 A CN 104572920A CN 201410836030 A CN201410836030 A CN 201410836030A CN 104572920 A CN104572920 A CN 104572920A
- Authority
- CN
- China
- Prior art keywords
- data
- expired
- tables
- key
- value pair
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/1737—Details of further file system functions for reducing power consumption or coping with limited storage space, e.g. in mobile devices
Abstract
An embodiment of the invention provides a data arrangement method and a data arrangement device. The data arrangement method comprises specific steps of judging whether a data sheet exceeds the time limit or not according to first expiration time information of the data sheet in a data arrangement process; and deleting data of all key value pairs stored in the data sheet when the data sheet exceeds the time limit. At least one key value pair datum is stored in the data sheet. By the embodiment, invalid data sheets can be recycled quickly and efficiently according to needs of business scenarios, so that storage space occupied by the invalid data sheet can be saved effectively.
Description
Technical field
The present invention relates to technical field of data storage, particularly relate to a kind of data preparation method and apparatus.
Background technology
Along with Internet era arrive, the Internet user be found everywhere through the world carries out alternately diversified every day, at any time all in the various data of manufacture, the quantity of these data is several times of unit epoch data volume, realizes high reliability, high performance storage also becomes particularly important to these data.
Such as, NoSQL is a kind of non-relational database, it stores with key-value pair (key-value), structure is not fixed, each tuple can have different field, each tuple can increase some oneself key-value pair as required, is not limited to fixing structure, therefore can reduces the expense of some Time and place.
Because user can produce a lot of gibberishes to the write operation of NoSQL, therefore, existing data preparation scheme carries out duplicate removal on the basis of data search, and such as corresponding to identical key multiple value data carry out union operation, to realize the recovery of these gibberishes.But some business scenario also can exist specific gibberish, existing data preparation scheme cannot carry out data preparation targetedly for business scenario.
Summary of the invention
In view of the above problems, the present invention is proposed to provide a kind of a kind of data preparation method and apparatus overcoming the problems referred to above or solve the problem at least in part.
According to one aspect of the present invention, provide a kind of data preparation method, comprising:
In data-handling procedure, the first expired time information according to tables of data judges that whether tables of data is expired, stores at least one key-value pair data in wherein said tables of data; And
When described tables of data is expired, deletion action is performed to all key-value pair data that described tables of data stores.
Alternatively, described method also comprises:
When described tables of data is not out of date, judge that whether described key-value pair data is expired according to the second expired time of key-value pair data in described tables of data;
When described key-value pair data is expired, deletion action is performed to described key-value pair data.
Alternatively, described first expired time information is the information be recorded in data base configuration file, or, the information set when described first expired time information is and creates described tables of data.
Alternatively, described second expired time information is the information produced when writing described key-value pair data.
Alternatively, described method also comprises:
Described tables of data and described key-value pair data all not out of date time, the key-value pair data stored described tables of data performs duplicate removal operation.
Alternatively, described method also comprises:
Receive the read request from client;
The first expired time information according to described read request corresponding data table judges that whether tables of data is expired; And
When described tables of data is expired, return the first failure information to client.
Alternatively, described method also comprises:
When described tables of data is not out of date, the first expired time information according to the corresponding key-value pair of described read request judges that whether described key-value pair data is expired;
When described key-value pair data is expired, return the second failure information to client.
According to another aspect of the present invention, provide a kind of data preparation device, comprising:
First judge module, in data-handling procedure, the first expired time information according to tables of data judges that whether tables of data is expired, stores at least one key-value pair data in wherein said tables of data; And
First removing module, for when described tables of data is expired, performs deletion action to all key-value pair data that described tables of data stores.
Alternatively, described device also comprises:
According to the second expired time of key-value pair data in described tables of data, second judge module, for when described tables of data is not out of date, judges that whether described key-value pair data is expired;
Second removing module, for when described key-value pair data is expired, performs deletion action to described key-value pair data.
Alternatively, described first expired time information is the information be recorded in data base configuration file, or, the information set when described first expired time information is and creates described tables of data.
According to a kind of data preparation method and apparatus of the embodiment of the present invention, can in data-handling procedure, the first expired time information according to tables of data judges that whether tables of data is expired, and when described tables of data is expired, deletion action is performed to all key-value pair data that described tables of data stores, therefore, it is possible to the demand of foundation business scenario realizes the recovery of invalid data table fast and efficiently, thus effectively can save the storage space shared by these invalid data table.
In addition, when described tables of data is not out of date, according to the second expired time of key-value pair data in described tables of data, the embodiment of the present invention can also judge that whether described key-value pair data is expired, and when described key-value pair data is expired, deletion action is performed to described key-value pair data, therefore the recovery of invalid data table can either be realized fast and efficiently according to the demand of business scenario, the recovery of invalid key-value pair data can also be realized according to the demand of business scenario neatly, thus effectively can save the storage space shared by these invalid datas.
Above-mentioned explanation is only the general introduction of technical solution of the present invention, in order to technological means of the present invention can be better understood, and can be implemented according to the content of instructions, and can become apparent, below especially exemplified by the specific embodiment of the present invention to allow above and other objects of the present invention, feature and advantage.
Accompanying drawing explanation
By reading the detailed description of hereafter Alternate embodiments, various other advantage and benefit will become cheer and bright for those of ordinary skill in the art.Accompanying drawing only for illustrating the object of Alternate embodiments, and does not think limitation of the present invention.And in whole accompanying drawing, represent identical parts by identical reference symbol.In the accompanying drawings:
Fig. 1 shows a kind of according to an embodiment of the invention steps flow chart schematic diagram of data preparation method;
Fig. 2 shows a kind of according to an embodiment of the invention steps flow chart schematic diagram of data preparation method;
Fig. 3 shows the structural representation of a kind of data-storage system according to the present invention's example;
Fig. 4 shows the steps flow chart schematic diagram of a kind of data preparation method according to the present invention's example;
Fig. 5 shows a kind of according to an embodiment of the invention steps flow chart schematic diagram of data access method;
Fig. 6 shows a kind of according to an embodiment of the invention steps flow chart schematic diagram of data access method;
Fig. 7 shows a kind of according to an embodiment of the invention structural representation of data preparation device;
Fig. 8 shows a kind of according to an embodiment of the invention structural representation of data preparation device;
Fig. 9 shows a kind of according to an embodiment of the invention structural representation of data preparation device; And
Figure 10 shows a kind of according to an embodiment of the invention structural representation of data preparation device.
Embodiment
Below with reference to accompanying drawings exemplary embodiment of the present disclosure is described in more detail.Although show exemplary embodiment of the present disclosure in accompanying drawing, however should be appreciated that can realize the disclosure in a variety of manners and not should limit by the embodiment set forth here.On the contrary, provide these embodiments to be in order to more thoroughly the disclosure can be understood, and complete for the scope of the present disclosure can be conveyed to those skilled in the art.
With reference to Fig. 1, show a kind of according to an embodiment of the invention steps flow chart schematic diagram of data preparation method, specifically can comprise the steps:
Step 101, in data-handling procedure, judge that whether tables of data is expired according to the first expired time information of tables of data, in wherein said tables of data, store at least one key-value pair data; And
Step 102, when described tables of data is expired, deletion actions are performed to all key-value pair data that described tables of data stores.
The data of the embodiment of the present invention can be the data of miscellaneous service scene.Existing data preparation scheme carries out duplicate removal on the basis of data search, and such as corresponding to identical key multiple value data carry out union operation.
For some business scenario, its data often have certain ageing.In order to realize ageing management, the embodiment of the present invention can arrange expired time for tables of data (table), like this, in data-handling procedure, can judge that whether tables of data is expired according to the first expired time information of tables of data, and when described tables of data is expired, perform deletion action to all key-value pair data that described tables of data stores, therefore, it is possible to the demand of foundation business scenario realizes the recovery of invalid data table fast and efficiently.
In a kind of embodiment of the present invention, first expired time information of described tables of data can for being recorded in the information in data base configuration file, also be, can the first expired time information parameter being set in the configuration file of database, being configured the first expired time information of all tables of data by user by revising described first expired time information parameter.
In another kind of embodiment of the present invention, the first expired time information of described tables of data for the information set when creating described tables of data, can have independently the first expired time to make different pieces of information table.
To sum up, the embodiment of the present invention is in data-handling procedure, can judge that whether tables of data is expired according to the first expired time information of tables of data, and when described tables of data is expired, deletion action is performed to all key-value pair data that described tables of data stores, therefore, it is possible to the demand of foundation business scenario realizes the recovery of invalid data table fast and efficiently.
With reference to Fig. 2, show a kind of according to an embodiment of the invention steps flow chart schematic diagram of data preparation method, specifically can comprise the steps:
Step 201, in data-handling procedure, judge that whether tables of data is expired according to the first expired time information of tables of data, in wherein said tables of data, store at least one key-value pair data;
Step 202, when described tables of data is expired, deletion actions are performed to all key-value pair data that described tables of data stores;
Step 203, when described tables of data is not out of date, judge that whether described key-value pair data is expired according to the second expired time of key-value pair data in described tables of data;
Step 204, when described key-value pair data is expired, deletion action is performed to described key-value pair data.
Relative to embodiment illustrated in fig. 1, the present embodiment can also arrange independently the second expired time for every bar key-value pair data, like this, when described tables of data is not out of date, can also judge that whether described key-value pair data is expired according to the second expired time of key-value pair data in described tables of data, and when described key-value pair data is expired, deletion action is performed to described key-value pair data, therefore can either be quick according to the demand of business scenario, realize the recovery of invalid data table efficiently, the recovery of invalid key-value pair data can also be realized neatly according to the demand of business scenario.
In a kind of embodiment of the present invention, described second expired time information can for the information set when writing described key-value pair data.In specific implementation, can provide the write interface arranging expired time for key-value pair data, this write interface is used in the expired time of setting data when writing key-value pair data.Such as, in a kind of application example of the present invention, while write interface set_with_ttl (key, value, ttl) interface is used in write key-value, specify the expired time ttl of key-value.When key-value is after ttl second, automatically will be deleted by data preparation flow process of the present invention.
In another kind of embodiment of the present invention, described method can also comprise: described tables of data and described key-value pair data all not out of date time, the key-value pair data stored described tables of data performs duplicate removal operation.Described duplicate removal operation mainly can comprise: merge many data of identical key, be inserted into data-storage system etc. to the data of different key, and the present invention is not limited the operation of concrete duplicate removal.
To sum up, the present embodiment is when described tables of data is not out of date, judge that whether described key-value pair data is expired according to the second expired time of key-value pair data in described tables of data, and when described key-value pair data is expired, deletion action is performed to described key-value pair data, therefore, it is possible to the demand of foundation business scenario realizes the recovery of key-value pair data neatly.
For making those skilled in the art understand the embodiment of the present invention better, provide the example of carrying out data preparation for data-storage system at this.
With reference to Fig. 3, show the structural representation of a kind of data-storage system according to the present invention's example, have and can comprise: internal memory 301 and disk 302;
Wherein, internal memory 301 may further include: memory table and read-only memory table;
Disk 302 may further include: the composition such as 0 layer, 1 layer, a 2 layers LSM (log-structured merging, Log Structured Merge) hierarchical structure set, these hierarchical structures can be used for storing daily record file, meta-information file, current file and data file;
Wherein, data file can end up by .sst, is used for storing data and index;
Journal file ends up with .log, is used for storing nearest Data Update operation; Adopt the form adding and write, each write operation is all appended to the end of journal file, the memory table that each journal file is corresponding current, write operation first writes journal file, then temporary file is upgraded, even if system crash like this, also can recover the memory table in internal memory from journal file, the loss of data can not be caused;
The data file that meta-information file lists each layer (level) is formed, the scope of the key of each data file and other metamessage; The same journal file of meta-information file form, other change (increase of data file and deletion etc.) operation of data file level can be appended to the end of meta-information file;
Current file is a simple text, and its content is the meta-information file name of current use.
In actual applications, when the data committed memory of memory table insertion is after a boundary, can the record of internal memory be exported in the data file of disk, then system can generate new journal file and memory table, original memory table just becomes read-only memory table, after memory table is written into data file, corresponding journal file can be deleted.
After the data file number under certain layer exceedes threshold value, data preparation flow process can be performed.Certainly, those skilled in the art can also regularly perform data preparation flow process according to the actual requirements, and the embodiment of the present invention is not limited the actual conditions performing data preparation flow process.
With reference to Fig. 4, show the steps flow chart schematic diagram of a kind of data preparation method according to the present invention's example, this example relates to and data file in L (L is more than or equal to 0) layer and L+1 layer data file being arranged, and specifically can comprise the steps:
Step 401, from L layer, select the first data file to be arranged;
In actual applications, the tables of data of operation layer splits into a lot of parts usually, the corresponding LSM tree of each part, then can the deletion action of executed in parallel each several part.Particularly, the first data file corresponding to each several part can being searched, then carrying out data preparation operation to searching the first data file obtained.
First expired time information of step 402, foundation tables of data judges that whether the data table data stored in the first data file is expired;
Step 403, when described tables of data is expired, deletion actions is performed to all key-value pair data that described tables of data stores, and performs step 406;
Step 404, when described tables of data is not out of date, judge that whether described key-value pair data is expired according to the second expired time of key-value pair data in described first data file;
Step 405, when described key-value pair data is expired, deletion action is performed to described key-value pair data, and performs step 406;
Step 406, from L+1 layer, select the second data file to be combined;
Step 407, size according to key, carry out merge sort to key-value pair data in described first data file and the second data file;
Step 408, foundation the second expired time, judge that in merge sort result, whether current key-value pair data is expired, if so, then perform step 409, otherwise perform 410;
Step 409, abandon this current key-value pair data in merge sort result;
Step 410, judge before L layer layer in whether there is this current key corresponding data in merge sort result, if so, then perform step 409, otherwise perform 411;
If for level lower than the record that there is same Key in the file of L, so illustrated for Key, have fresher Value to exist, the Value so pass by just is equal to nothing meaning, so can abandon.
Step 411, by data file newly-generated in this current key-value pair data write L+1 layer in merge sort result.
Can find out, this example key-value pair data in described first data file and the second data file has deleted data expired in described first data file before carrying out merge sort, therefore, it is possible to reduce the data volume of merge sort;
This example is before writing data file newly-generated in L+1 layer by key-value pair data in merge sort result, abandon key-value pair data expired in merge sort result, also be, what data file newly-generated in L+1 layer write is not out of date key-value pair data, therefore, it is possible to effectively save the storage space of disk.
It should be noted that, above-mentioned in described first data file and the second data file key-value pair data carry out merge sort and carry out the mode of expired deletion just exemplarily by before data file newly-generated in key-value pair data write L+1 layer in merge sort result, those skilled in the art can carry out expired deletion action according to the either phase of actual demand in data preparation to tables of data or data, and the concrete stage of the embodiment of the present invention to expired deletion is not limited.
With reference to Fig. 5, show a kind of according to an embodiment of the invention steps flow chart schematic diagram of data preparation method, specifically can comprise the steps:
Step 501, in data-handling procedure, judge that whether tables of data is expired according to the first expired time information of tables of data, in wherein said tables of data, store at least one key-value pair data;
Step 502, when described tables of data is expired, deletion actions are performed to all key-value pair data that described tables of data stores;
Step 503, receive read request from client;
Step 504, judge that whether tables of data is expired according to the first expired time information of described read request corresponding data table; And
Step 505, when described tables of data is expired, return the first failure information to client.
Relative to embodiment illustrated in fig. 1, the present embodiment can also have the function of process read request, wherein, in the process of process read request, can judge that whether tables of data is expired according to the first expired time information of described read request corresponding data table, if the tables of data at the corresponding key place of described read request is expired, reclaim even if this tables of data is not arranged flow process by aforementioned data, still the first failure information is returned to client, with the demand making data access flow process adapt to business scenario.In actual applications, information such as " user data do not exist " can be carried in described first failure information.
It should be noted that, when described tables of data is not out of date, key-value pair data corresponding to described read request can be returned to client.
With reference to Fig. 6, show a kind of according to an embodiment of the invention steps flow chart schematic diagram of data access method, specifically can comprise the steps:
Step 601, in data-handling procedure, judge that whether tables of data is expired according to the first expired time information of tables of data, in wherein said tables of data, store at least one key-value pair data;
Step 602, when described tables of data is expired, deletion actions are performed to all key-value pair data that described tables of data stores;
Step 603, receive read request from client;
Step 604, judge that whether tables of data is expired according to the first expired time information of described read request corresponding data table;
Step 605, when described tables of data is expired, return the first failure information to client;
Step 606, when described tables of data is not out of date, the first expired time information according to the corresponding key-value pair of described read request judges that whether described key-value pair data is expired;
Step 607, when described key-value pair data is expired, return the second failure information to client.
Relative to embodiment illustrated in fig. 5, the present embodiment is in the process of process read request, can also judge that whether described key-value pair data is expired according to the first expired time information of the corresponding key-value pair of described read request, if corresponding key is expired for described read request, even if this key is not arranged flow process by aforementioned data and reclaims, still the second failure information is returned to client, with the demand making data access flow process adapt to business scenario.In actual applications, information such as " user data do not exist " can be carried in described second failure information.
For embodiment of the method, in order to simple description, therefore it is all expressed as a series of combination of actions, but those skilled in the art should know, the embodiment of the present invention is not by the restriction of described sequence of movement, because according to the embodiment of the present invention, some step can adopt other orders or carry out simultaneously.Secondly, those skilled in the art also should know, the embodiment described in instructions all belongs to embodiment, and involved action might not be that the embodiment of the present invention is necessary.
With reference to Fig. 7, show a kind of according to an embodiment of the invention structural representation of data preparation device, specifically can comprise as lower module:
First judge module 701, in data-handling procedure, the first expired time information according to tables of data judges that whether tables of data is expired, stores at least one key-value pair data in wherein said tables of data; And
First removing module 702, for when described tables of data is expired, performs deletion action to all key-value pair data that described tables of data stores.
In a kind of embodiment of the present invention, described first expired time information can for being recorded in the information in data base configuration file, or described first expired time information can for the information set when creating described tables of data.
With reference to Fig. 8, show a kind of according to an embodiment of the invention structural representation of data preparation device, specifically can comprise as lower module:
First judge module 801, in data-handling procedure, the first expired time information according to tables of data judges that whether tables of data is expired, stores at least one key-value pair data in wherein said tables of data;
First removing module 802, for when described tables of data is expired, performs deletion action to all key-value pair data that described tables of data stores;
According to the second expired time of key-value pair data in described tables of data, second judge module 803, for when described tables of data is not out of date, judges that whether described key-value pair data is expired; And
Second removing module 804, for when described key-value pair data is expired, performs deletion action to described key-value pair data.
In a kind of embodiment of the present invention, described second expired time information can for the information set when writing described key-value pair data.
With reference to Fig. 9, show a kind of according to an embodiment of the invention structural representation of data preparation device, specifically can comprise as lower module:
First judge module 901, in data-handling procedure, the first expired time information according to tables of data judges that whether tables of data is expired, stores at least one key-value pair data in wherein said tables of data;
First removing module 902, for when described tables of data is expired, performs deletion action to all key-value pair data that described tables of data stores;
Receiver module 903, for receiving the read request from client;
According to the first expired time information of described read request corresponding data table, 3rd judge module 904, for judging that whether tables of data is expired; And
First returns module 905, for when described tables of data is expired, returns the first failure information to client.
With reference to Figure 10, show a kind of according to an embodiment of the invention structured flowchart of data preparation device, specifically can comprise as lower module:
First judge module 1001, in data-handling procedure, the first expired time information according to tables of data judges that whether tables of data is expired, stores at least one key-value pair data in wherein said tables of data;
First removing module 1002, for when described tables of data is expired, performs deletion action to all key-value pair data that described tables of data stores;
Receiver module 1003, for receiving the read request from client;
According to the first expired time information of described read request corresponding data table, 3rd judge module 1004, for judging that whether tables of data is expired;
First returns module 1005, for when described tables of data is expired, returns the first failure information to client;
4th judge module 1006, for when described tables of data is not out of date, the first expired time information according to the corresponding key-value pair of described read request judges that whether described key-value pair data is expired; And
Second returns module 1007, for when described key-value pair data is expired, returns the second failure information to client.
For device embodiment, due to itself and embodiment of the method basic simlarity, so description is fairly simple, relevant part illustrates see the part of embodiment of the method.
Intrinsic not relevant to any certain computer, virtual system or miscellaneous equipment with display at this algorithm provided.Various general-purpose system also can with use based on together with this teaching.According to description above, the structure constructed required by this type systematic is apparent.In addition, the present invention is not also for any certain programmed language.It should be understood that and various programming language can be utilized to realize content of the present invention described here, and the description done language-specific is above to disclose preferred forms of the present invention.
In instructions provided herein, describe a large amount of detail.But can understand, embodiments of the invention can be put into practice when not having these details.In some instances, be not shown specifically known method, structure and technology, so that not fuzzy understanding of this description.
Similarly, be to be understood that, in order to simplify the disclosure and to help to understand in each inventive aspect one or more, in the description above to exemplary embodiment of the present invention, each feature of the present invention is grouped together in single embodiment, figure or the description to it sometimes.But, the method for the disclosure should be construed to the following intention of reflection: namely the present invention for required protection requires feature more more than the feature clearly recorded in each claim.Or rather, as claims below reflect, all features of disclosed single embodiment before inventive aspect is to be less than.Therefore, the claims following embodiment are incorporated to this embodiment thus clearly, and wherein each claim itself is as independent embodiment of the present invention.
Those skilled in the art are appreciated that and adaptively can change the module in the equipment in embodiment and they are arranged in one or more equipment different from this embodiment.Module in embodiment or unit or assembly can be combined into a module or unit or assembly, and multiple submodule or subelement or sub-component can be put them in addition.Except at least some in such feature and/or process or unit be mutually repel except, any combination can be adopted to combine all processes of all features disclosed in this instructions (comprising adjoint claim, summary and accompanying drawing) and so disclosed any method or equipment or unit.Unless expressly stated otherwise, each feature disclosed in this instructions (comprising adjoint claim, summary and accompanying drawing) can by providing identical, alternative features that is equivalent or similar object replaces.
In addition, those skilled in the art can understand, although embodiments more described herein to comprise in other embodiment some included feature instead of further feature, the combination of the feature of different embodiment means and to be within scope of the present invention and to form different embodiments.Such as, in the following claims, the one of any of embodiment required for protection can use with arbitrary array mode.
All parts embodiment of the present invention with hardware implementing, or can realize with the software module run on one or more processor, or realizes with their combination.It will be understood by those of skill in the art that the some or all functions that microprocessor or digital signal processor (DSP) can be used in practice to realize according to the some or all parts in the data preparation method and apparatus of the embodiment of the present invention.The present invention can also be embodied as part or all equipment for performing method as described herein or device program (such as, computer program and computer program).Realizing program of the present invention and can store on a computer-readable medium like this, or the form of one or more signal can be had.Such signal can be downloaded from Internet platform and obtain, or provides on carrier signal, or provides with any other form.
The present invention will be described instead of limit the invention to it should be noted above-described embodiment, and those skilled in the art can design alternative embodiment when not departing from the scope of claims.In the claims, any reference symbol between bracket should be configured to limitations on claims.Word " comprises " not to be got rid of existence and does not arrange element in the claims or step.Word "a" or "an" before being positioned at element is not got rid of and be there is multiple such element.The present invention can by means of including the hardware of some different elements and realizing by means of the computing machine of suitably programming.In the unit claim listing some devices, several in these devices can be carry out imbody by same hardware branch.Word first, second and third-class use do not represent any order.Can be title by these word explanations.
The invention discloses A1, a kind of data preparation method, comprising:
In data-handling procedure, the first expired time information according to tables of data judges that whether tables of data is expired, stores at least one key-value pair data in wherein said tables of data; And
When described tables of data is expired, deletion action is performed to all key-value pair data that described tables of data stores.
A2, method as described in A1, described method also comprises:
When described tables of data is not out of date, judge that whether described key-value pair data is expired according to the second expired time of key-value pair data in described tables of data;
When described key-value pair data is expired, deletion action is performed to described key-value pair data.
A3, method as described in A1 or A2, described first expired time information is the information be recorded in data base configuration file, or, the information set when described first expired time information is and creates described tables of data.
A4, method as described in A2, described second expired time information is the information produced when writing described key-value pair data.
A5, method as described in A2, described method also comprises:
Described tables of data and described key-value pair data all not out of date time, the key-value pair data stored described tables of data performs duplicate removal operation.
A6, method as described in A1 or A2, described method also comprises:
Receive the read request from client;
The first expired time information according to described read request corresponding data table judges that whether tables of data is expired; And
When described tables of data is expired, return the first failure information to client.
A7, method as described in A6, described method also comprises:
When described tables of data is not out of date, the first expired time information according to the corresponding key-value pair of described read request judges that whether described key-value pair data is expired;
When described key-value pair data is expired, return the second failure information to client.
The invention also discloses B8, a kind of data preparation device, comprising:
First judge module, in data-handling procedure, the first expired time information according to tables of data judges that whether tables of data is expired, stores at least one key-value pair data in wherein said tables of data; And
First removing module, for when described tables of data is expired, performs deletion action to all key-value pair data that described tables of data stores.
B9, device as described in B8, described device also comprises:
According to the second expired time of key-value pair data in described tables of data, second judge module, for when described tables of data is not out of date, judges that whether described key-value pair data is expired;
Second removing module, for when described key-value pair data is expired, performs deletion action to described key-value pair data.
B10, device as described in B8 or B9, described first expired time information is the information be recorded in data base configuration file, or, the information set when described first expired time information is and creates described tables of data.
B11, device as described in B9, described second expired time information is the information set when writing described key-value pair data.
B12, device as described in B9, described device also comprises:
Described tables of data and described key-value pair data all not out of date time, the key-value pair data stored described tables of data performs duplicate removal operation.
B13, device as described in B8 or B9, described device also comprises:
Receiver module, for receiving the read request from client;
According to the first expired time information of described read request corresponding data table, 3rd judge module, for judging that whether tables of data is expired;
First returns module, for when described tables of data is expired, returns the first failure information to client.
B14, device as described in B13, described device also comprises:
4th judge module, for when described tables of data is not out of date, the first expired time information according to the corresponding key-value pair of described read request judges that whether described key-value pair data is expired;
Second returns module, for when described key-value pair data is expired, returns the second failure information to client.
Claims (10)
1. a data preparation method, comprising:
In data-handling procedure, the first expired time information according to tables of data judges that whether tables of data is expired, stores at least one key-value pair data in wherein said tables of data; And
When described tables of data is expired, deletion action is performed to all key-value pair data that described tables of data stores.
2. the method for claim 1, is characterized in that, described method also comprises:
When described tables of data is not out of date, judge that whether described key-value pair data is expired according to the second expired time of key-value pair data in described tables of data;
When described key-value pair data is expired, deletion action is performed to described key-value pair data.
3. method as claimed in claim 1 or 2, it is characterized in that, described first expired time information is the information be recorded in data base configuration file, or, the information set when described first expired time information is and creates described tables of data.
4. method as claimed in claim 2, is characterized in that, described second expired time information is the information produced when writing described key-value pair data.
5. method as claimed in claim 2, it is characterized in that, described method also comprises:
Described tables of data and described key-value pair data all not out of date time, the key-value pair data stored described tables of data performs duplicate removal operation.
6. method as claimed in claim 1 or 2, it is characterized in that, described method also comprises:
Receive the read request from client;
The first expired time information according to described read request corresponding data table judges that whether tables of data is expired; And
When described tables of data is expired, return the first failure information to client.
7. method as claimed in claim 6, it is characterized in that, described method also comprises:
When described tables of data is not out of date, the first expired time information according to the corresponding key-value pair of described read request judges that whether described key-value pair data is expired;
When described key-value pair data is expired, return the second failure information to client.
8. a data preparation device, comprising:
First judge module, in data-handling procedure, the first expired time information according to tables of data judges that whether tables of data is expired, stores at least one key-value pair data in wherein said tables of data; And
First removing module, for when described tables of data is expired, performs deletion action to all key-value pair data that described tables of data stores.
9. device as claimed in claim 8, it is characterized in that, described device also comprises:
According to the second expired time of key-value pair data in described tables of data, second judge module, for when described tables of data is not out of date, judges that whether described key-value pair data is expired;
Second removing module, for when described key-value pair data is expired, performs deletion action to described key-value pair data.
10. device as claimed in claim 8 or 9, it is characterized in that, described first expired time information is the information be recorded in data base configuration file, or, the information set when described first expired time information is and creates described tables of data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410836030.6A CN104572920A (en) | 2014-12-27 | 2014-12-27 | Data arrangement method and data arrangement device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410836030.6A CN104572920A (en) | 2014-12-27 | 2014-12-27 | Data arrangement method and data arrangement device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN104572920A true CN104572920A (en) | 2015-04-29 |
Family
ID=53088982
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410836030.6A Pending CN104572920A (en) | 2014-12-27 | 2014-12-27 | Data arrangement method and data arrangement device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104572920A (en) |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105138622A (en) * | 2015-08-14 | 2015-12-09 | 中国科学院计算技术研究所 | Append operation method for LSM tree memory system and reading and merging method for loads of append operation |
| CN105447167A (en) * | 2015-12-04 | 2016-03-30 | 北京奇虎科技有限公司 | Processing method and apparatus for node cache data in distributed system |
| CN105488167A (en) * | 2015-11-30 | 2016-04-13 | 北京金山安全软件有限公司 | Index database updating method and device |
| CN105550299A (en) * | 2015-12-14 | 2016-05-04 | 镇江乐游网络科技有限公司 | Game user data management method |
| CN105574196A (en) * | 2015-12-25 | 2016-05-11 | 北京奇虎科技有限公司 | Database management method and device |
| CN106503260A (en) * | 2016-11-18 | 2017-03-15 | 北京奇虎科技有限公司 | A kind of method and apparatus of the effective memory space for improving data base |
| CN106681837A (en) * | 2016-12-29 | 2017-05-17 | 北京奇虎科技有限公司 | Data sheet based data eliminating method and device |
| CN107451190A (en) * | 2017-06-26 | 2017-12-08 | 北京五八信息技术有限公司 | Can persistence non-relational database data processing method and device |
| CN108196792A (en) * | 2017-12-29 | 2018-06-22 | 北京奇虎科技有限公司 | Remove the method and device of stale data |
| CN108319602A (en) * | 2017-01-17 | 2018-07-24 | 广州市动景计算机科技有限公司 | Data base management method and Database Systems |
| CN109032505A (en) * | 2018-06-26 | 2018-12-18 | 深圳忆联信息系统有限公司 | Data read-write method, device, computer equipment and storage medium with timeliness |
| CN110413679A (en) * | 2019-07-31 | 2019-11-05 | 深圳前海微众银行股份有限公司 | Database information processing method, device, equipment and readable storage medium storing program for executing |
| CN112214503A (en) * | 2020-10-10 | 2021-01-12 | 深圳壹账通智能科技有限公司 | Data processing method and device, electronic equipment and storage medium |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101556589A (en) * | 2008-04-09 | 2009-10-14 | 北京闻言科技有限公司 | Method for Oracle regularly deleting stale data in database |
| CN102043726A (en) * | 2010-12-29 | 2011-05-04 | 北京播思软件技术有限公司 | Storage management method of large-scale timing sequence data |
| US7958097B1 (en) * | 2003-09-24 | 2011-06-07 | Parallels Holdings, Ltd. | Method and system for implementation of data storage quota |
| CN102110146A (en) * | 2011-02-16 | 2011-06-29 | 清华大学 | Key-value storage-based distributed file system metadata management method |
| CN103207919A (en) * | 2013-04-26 | 2013-07-17 | 北京亿赞普网络技术有限公司 | Method and device for quickly inquiring and calculating MangoDB cluster |
| CN103345423A (en) * | 2013-07-03 | 2013-10-09 | 北京京东尚科信息技术有限公司 | Method and system for processing asynchronous task |
| CN104077343A (en) * | 2013-12-26 | 2014-10-01 | 国家计算机网络与信息安全管理中心 | Method for deleting invalid elements of hash table |
| CN104182436A (en) * | 2014-02-25 | 2014-12-03 | 无锡天脉聚源传媒科技有限公司 | Method and device for cleaning databases |
-
2014
- 2014-12-27 CN CN201410836030.6A patent/CN104572920A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7958097B1 (en) * | 2003-09-24 | 2011-06-07 | Parallels Holdings, Ltd. | Method and system for implementation of data storage quota |
| CN101556589A (en) * | 2008-04-09 | 2009-10-14 | 北京闻言科技有限公司 | Method for Oracle regularly deleting stale data in database |
| CN102043726A (en) * | 2010-12-29 | 2011-05-04 | 北京播思软件技术有限公司 | Storage management method of large-scale timing sequence data |
| CN102110146A (en) * | 2011-02-16 | 2011-06-29 | 清华大学 | Key-value storage-based distributed file system metadata management method |
| CN103207919A (en) * | 2013-04-26 | 2013-07-17 | 北京亿赞普网络技术有限公司 | Method and device for quickly inquiring and calculating MangoDB cluster |
| CN103345423A (en) * | 2013-07-03 | 2013-10-09 | 北京京东尚科信息技术有限公司 | Method and system for processing asynchronous task |
| CN104077343A (en) * | 2013-12-26 | 2014-10-01 | 国家计算机网络与信息安全管理中心 | Method for deleting invalid elements of hash table |
| CN104182436A (en) * | 2014-02-25 | 2014-12-03 | 无锡天脉聚源传媒科技有限公司 | Method and device for cleaning databases |
Cited By (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105138622A (en) * | 2015-08-14 | 2015-12-09 | 中国科学院计算技术研究所 | Append operation method for LSM tree memory system and reading and merging method for loads of append operation |
| CN105138622B (en) * | 2015-08-14 | 2018-05-22 | 中国科学院计算技术研究所 | For the insertion operation of LSM tree storage systems and reading and the merging method of load |
| CN105488167A (en) * | 2015-11-30 | 2016-04-13 | 北京金山安全软件有限公司 | Index database updating method and device |
| CN105488167B (en) * | 2015-11-30 | 2019-12-13 | 北京金山安全软件有限公司 | Index database updating method and device |
| CN105447167A (en) * | 2015-12-04 | 2016-03-30 | 北京奇虎科技有限公司 | Processing method and apparatus for node cache data in distributed system |
| CN105447167B (en) * | 2015-12-04 | 2019-03-01 | 北京奇虎科技有限公司 | The data cached processing method and processing device of distributed system interior joint |
| CN105550299A (en) * | 2015-12-14 | 2016-05-04 | 镇江乐游网络科技有限公司 | Game user data management method |
| CN105574196B (en) * | 2015-12-25 | 2019-02-26 | 北京奇虎科技有限公司 | A kind of data base management method and device |
| CN105574196A (en) * | 2015-12-25 | 2016-05-11 | 北京奇虎科技有限公司 | Database management method and device |
| CN106503260A (en) * | 2016-11-18 | 2017-03-15 | 北京奇虎科技有限公司 | A kind of method and apparatus of the effective memory space for improving data base |
| CN106503260B (en) * | 2016-11-18 | 2020-04-28 | 北京奇虎科技有限公司 | Method and device for improving effective storage space of database |
| CN106681837A (en) * | 2016-12-29 | 2017-05-17 | 北京奇虎科技有限公司 | Data sheet based data eliminating method and device |
| CN106681837B (en) * | 2016-12-29 | 2020-10-16 | 北京奇虎科技有限公司 | Data elimination method and device based on data table |
| CN108319602A (en) * | 2017-01-17 | 2018-07-24 | 广州市动景计算机科技有限公司 | Data base management method and Database Systems |
| CN108319602B (en) * | 2017-01-17 | 2020-10-16 | 阿里巴巴（中国）有限公司 | Database management method and database system |
| CN107451190A (en) * | 2017-06-26 | 2017-12-08 | 北京五八信息技术有限公司 | Can persistence non-relational database data processing method and device |
| CN108196792A (en) * | 2017-12-29 | 2018-06-22 | 北京奇虎科技有限公司 | Remove the method and device of stale data |
| CN109032505A (en) * | 2018-06-26 | 2018-12-18 | 深圳忆联信息系统有限公司 | Data read-write method, device, computer equipment and storage medium with timeliness |
| CN110413679A (en) * | 2019-07-31 | 2019-11-05 | 深圳前海微众银行股份有限公司 | Database information processing method, device, equipment and readable storage medium storing program for executing |
| CN112214503A (en) * | 2020-10-10 | 2021-01-12 | 深圳壹账通智能科技有限公司 | Data processing method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104572920A (en) | Data arrangement method and data arrangement device | |
| JP7410181B2 (en) | Hybrid indexing methods, systems, and programs | |
| CN103678494B (en) | Client synchronization services the method and device of end data | |
| US8924365B2 (en) | System and method for range search over distributive storage systems | |
| CN109416694A (en) | The key assignments storage system effectively indexed including resource | |
| CN103164490B (en) | A kind of efficient storage implementation method of not fixed-length data and device | |
| US10733045B2 (en) | Online repair of metadata for structured data including file systems | |
| CN103902698A (en) | Data storage system and data storage method | |
| CN105468473A (en) | Data migration method and data migration apparatus | |
| CN108021717B (en) | Method for implementing lightweight embedded file system | |
| US9245003B2 (en) | Method and system for memory efficient, update optimized, transactional full-text index view maintenance | |
| CN105550306A (en) | Multi-copy data reading/writing method and system | |
| CN103902702A (en) | Data storage system and data storage method | |
| CN105095261A (en) | Data insertion method and device | |
| CN110888837B (en) | Object storage small file merging method and device | |
| WO2018046085A1 (en) | Systems and methods for performing a range query on a skiplist data structure | |
| CN103186622A (en) | Updating method of index information in full text retrieval system and device thereof | |
| Petrov | Database Internals: A deep dive into how distributed data systems work | |
| CN109690522B (en) | Data updating method and device based on B+ tree index and storage device | |
| CN111104377A (en) | File management method, electronic device and computer-readable storage medium | |
| CN107766355B (en) | Hierarchical data management method, hierarchical data management system and instant messaging system | |
| CN102902770A (en) | Method and system for assembling mirror image files | |
| CN100565495C (en) | File index is handled | |
| CN103177026A (en) | Data management method and data management system | |
| US9367573B1 (en) | Methods and apparatus for archiving system having enhanced processing efficiency |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20150429 |