CN105446893A - Data storage method and device - Google Patents
Data storage method and device Download PDFInfo
- Publication number
- CN105446893A CN105446893A CN201410333794.3A CN201410333794A CN105446893A CN 105446893 A CN105446893 A CN 105446893A CN 201410333794 A CN201410333794 A CN 201410333794A CN 105446893 A CN105446893 A CN 105446893A
- Authority
- CN
- China
- Prior art keywords
- data
- local cache
- stored
- described local
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a data storage method and device; the data storage method comprises the following steps: storing to be stored data into a local buffer memory and/or a database according to a first data storage logic when the to be stored data arrives; storing the data of the local buffer memory into the database according to a second data storage logic when a preset synchronization period arrives. The method and device can ensure data storage device throughput, and can improve data storage instantaneity.
Description
[technical field]
The present invention relates to technical field of data storage, particularly relate to a kind of date storage method and equipment.
[background technology]
Hbase be one distributed, towards row PostgreSQL database.As the client of Hbase, first can write data into local cache when writing data to Hbase, after the data volume in local cache exceedes threshold value, then by the data one-time write Hbase in local cache, the handling capacity of client can be ensured like this.But because data first write the local cache of client, just can be written into Hbase after only having the data volume in local cache to exceed threshold value, therefore, real-time is very poor.
[summary of the invention]
Many aspects of the present invention provide a kind of date storage method and equipment, in order to while the handling capacity ensureing data storage device, improve the real-time that data store.
An aspect of of the present present invention, provides a kind of date storage method, comprising:
When there being data to be stored to arrive, according to the first data storage logic, described data to be stored are stored in local cache and/or database;
When arriving the synchronizing cycle of presetting, according to the second data storage logic, the data in described local cache are stored in described database.
Another aspect of the present invention, provides a kind of data storage device, comprising:
Described data to be stored, for when there being data to be stored to arrive, are stored in local cache and/or database according to the first data storage logic by memory module;
Data in described local cache, during for arriving in the synchronizing cycle of presetting, are stored in described database according to the second data storage logic by synchronization module.
In technical solution of the present invention, when there being data to be stored to arrive, according to the first data storage logic, data to be stored are stored in local cache and/or database, wherein, if first data to be stored are deposited local cache, be conducive to the handling capacity improving data storage device, if be directly stored in database by data to be stored, be then conducive to the real-time improving data storage; In addition, also synchronizing cycle is set, when arriving synchronizing cycle, according to the second data storage logic, the data in local cache are stored in database, can prevent data in local cache for a long time cannot the situation of write into Databasce, improve real-time data be stored in database to a certain extent.
[accompanying drawing explanation]
In order to be illustrated more clearly in the technical scheme in the embodiment of the present invention, be briefly described to the accompanying drawing used required in embodiment or description of the prior art below, apparently, accompanying drawing in the following describes is some embodiments of the present invention, for those of ordinary skill in the art, under the prerequisite not paying creative work, other accompanying drawing can also be obtained according to these accompanying drawings.
The schematic flow sheet of the date storage method that Fig. 1 provides for one embodiment of the invention;
The schematic flow sheet of the date storage method that Fig. 2 provides for another embodiment of the present invention;
The structural representation of the data storage device that Fig. 3 provides for one embodiment of the invention;
The structural representation of the data storage device that Fig. 4 provides for another embodiment of the present invention.
[embodiment]
For making the object of the embodiment of the present invention, technical scheme and advantage clearly, below in conjunction with the accompanying drawing in the embodiment of the present invention, technical scheme in the embodiment of the present invention is clearly and completely described, obviously, described embodiment is the present invention's part embodiment, instead of whole embodiments.Based on the embodiment in the present invention, those of ordinary skill in the art, not making the every other embodiment obtained under creative work prerequisite, belong to the scope of protection of the invention.
The schematic flow sheet of the date storage method that Fig. 1 provides for one embodiment of the invention.As shown in Figure 1, the method comprises:
101, when there being data to be stored to arrive, according to the first data storage logic, these data to be stored are stored in local cache and/or database.
102, when arriving the synchronizing cycle of presetting, according to the second data storage logic, the data in local cache are stored in database.
First what deserves to be explained is not have sequencing between above-mentioned steps 101 and 102, which condition first arrives just first performs operation corresponding to which condition.Specifically, if first there are data to be stored to arrive, then first according to the first data storage logic, data to be stored are stored in local cache and/or database, afterwards, when arriving the synchronizing cycle of presetting, then according to the second data storage logic, the data in local cache are stored in database.If first arrive the synchronizing cycle of presetting, then first according to the second data storage logic, the data in local cache are stored in database, afterwards, when there being data to be stored to arrive, then according to the first data storage logic, data to be stored are stored in local cache and/or database.
The present embodiment does not limit above-mentioned first data storage logic, everyly the data storage logic that data to be stored are stored in local cache and/or database all can be applicable to the present invention.In like manner, the present embodiment does not also limit above-mentioned second data storage logic, everyly when synchronizing cycle arrives, the data storage logic that the data in local cache are stored in database also all can be applicable to the present invention.
In the present embodiment, when there being data to be stored to arrive, data to be stored can be stored in local cache and/or database according to the first data storage logic by data storage device.Local cache refers to the spatial cache of data storage device side.
Illustrate at this, data to be stored can, when data to be stored arrive, directly be stored in database by data storage device, are stored in database by data to be stored namely in real time, can ensure the real-time that data store like this.Or, when data storage device can have a storage space in local cache, data to be stored are stored in local cache, and when there is no storage space in local cache, data to be stored are stored in database.Such elder generation carries out buffer memory in local cache, and the data in local cache can be unified afterwards to be stored in database, is conducive to the handling capacity improving data storage device.
In addition, in the present embodiment, pre-set synchronizing cycle, according to the second data storage logic, the data in local cache are stored in database for periodic, to avoid the data in local cache not to be written to for a long time in database, improve the real-time that data are stored in database.In order to the beneficial effect of the real-time that understanding the present embodiment clearly brings, below to the data in local cache in prior art for a long time situation about not being written in database be illustrated:
Suppose that the size of local cache is 5 spatial caches, each data take 1 spatial cache, there are 2 secondary data write requests front and back, the data of the first write request request write are the first data, the second data and the 3rd data, and the data of the second write request request write are the 4th data, the 5th data, the 6th data and the 7th data;
When process the first write request, the space of local cache is enough, and now, the data in local cache are the first data, the second data and the 3rd data, temporarily do not have data in database;
When process the second write request, the insufficient space of local cache, after write the 4th data and the 5th data, data volume in local cache has reached threshold value (referring to that local cache is full in this embodiment), then need in the data one-time write database in local cache, then empty local cache, then continue write the 6th data and the 7th data.Because the space of now local cache is enough, the 6th data then in local cache and the 7th data will not be written into database, if the data volume in local cache does not reach threshold value (i.e. local cache less than) always, then the 6th data and the 7th data will not be written into database always, cause the 6th data and the 7th data to be stored in database in time, real-time is poor.
And the present embodiment is by arranging synchronizing cycle, when arriving synchronizing cycle, whether the data volume no matter in local cache reaches threshold value, forces to be stored in database according to the second data storage logic by the data in local cache, is conducive to ensureing that data are stored in the real-time in database.Such as, when arriving synchronizing cycle, the 6th data in local cache and the 7th data just can be stored in database by data storage device, ensure that the 6th data and the 7th data can be stored in database in time, improve the real-time that data store.
Illustrate at this, above-mentioned synchronizing cycle can according to practical application adaptability equipment, and the value of the present embodiment to synchronizing cycle does not limit, such as, can be 30 seconds, 50 seconds, 1 minute, 5 minutes, 10 minutes, 30 minutes, 1 hour, 1 day etc.
The database related in the embodiment of the present invention can be various database, can be Hbase.
Below to the first data storage logic, namely the implementation of step 101 is described.
A kind of implementation of step 101 is: when there being data to be stored to arrive, data storage device judges whether the data volume in local cache is greater than default first threshold, first threshold that if the data volume in local cache is not more than (being namely less than or equal to), then by data write local cache to be stored; If the data volume in local cache is greater than first threshold, then by the data write into Databasce in local cache, and by data write local cache to be stored after emptying local cache.In this implementation, when local cache has storage space, first data to be stored are stored into local cache, after the data volume in local cache reaches first threshold, data in local cache are stored in database by property again, are conducive to the handling capacity improving data storage device.
The another kind of implementation of step 101 comprises: when there being data to be stored to arrive, data storage device is by data write local cache to be stored, after by data write local cache to be stored, judge whether the data volume in local cache is greater than default first threshold, if the data volume of local cache is greater than first threshold, then by the data write into Databasce in local cache, and empty local cache; If the data volume in local cache is not more than first threshold, then terminate this time to store operation.It is different with the sequencing whether data volume judged in local cache is greater than first threshold that this implementation and the difference of above-mentioned implementation are to store data to be stored, but technical scheme is substantially identical.Therefore, this implementation is also conducive to the handling capacity improving data storage device.
Another implementation of step 101 comprises: when there being data to be stored to arrive, data storage device judges whether the data volume in local cache is greater than default first threshold, if the data volume in local cache is not more than first threshold, then by data write local cache to be stored; If the data volume in local cache is greater than first threshold, then by the data in data to be stored and local cache in the lump write into Databasce, and empty local cache.In this implementation, when local cache has storage space, first data to be stored are stored into local cache, after the data volume in local cache reaches first threshold, data in local cache are stored in database together with data to be stored by property again, be conducive to the handling capacity improving data storage device, also help simultaneously and improve the real-time of data write into Databasce to be stored.
Below to the second data storage logic, namely the implementation of step 102 is described.
A kind of implementation of step 102 comprises: when arriving the synchronizing cycle of presetting, data storage device judges whether have data in local cache, if having data in local cache, then by the data write into Databasce in local cache, and empties local cache.If do not have data in local cache, then data storage device can not perform any operation.In this implementation, when arriving synchronizing cycle, as long as have data in local cache, just by all data one-time write databases, can ensure that in local cache, all data are written into the real-time in database.
The another kind of implementation of step 102 comprises: when arriving the synchronizing cycle of presetting, data storage device judges whether have data in local cache, if have data in local cache, then by the storage time in local cache of N number of data write into Databasce the earliest, and N number of data of write into Databasce are deleted from local cache, N is the natural number of the data volume be less than in local cache.If do not have data in local cache, then data storage device can not perform any operation.In this implementation, when arriving synchronizing cycle, data storage device sorted to the data in local cache according to storage time, by oldest stored to the partial data in local cache, namely storage time N number of data the earliest, in write into Databasce, instead of by all data in local cache all write into Databasce, such one side can ensure that the data write the earliest in local cache are written into the real-time in database, is conducive on the other hand realizing the equilibrium between two kinds of operations.Described two kinds of operations refer to that factor data amount is greater than first threshold by the operation of the data write into Databasce in local cache with because arriving the operation of the data write into Databasce in local cache synchronizing cycle.
Another implementation of step 102 comprises: when arriving the synchronizing cycle of presetting, data storage device judges whether have data in local cache, if have data in local cache, then judge whether the data volume in local cache is greater than default Second Threshold, if the data volume in local cache is greater than Second Threshold, then by the data write into Databasce in local cache, and empty local cache.If the data volume in local cache is not more than Second Threshold, or does not have data in local cache, then data storage device can not perform any operation.Preferably, Second Threshold is less than first threshold.In this implementation, by arranging Second Threshold, only have when the data volume in local cache is greater than Second Threshold, data storage device just can by the data write into Databasce in local cache, be conducive to reducing the interaction times with database, be conducive to improving the handling capacity of data storage device, the data that simultaneously can ensure again in local cache are written into the real-time in database.
Illustrate at this, several implementation of above-mentioned steps 101 and several implementations of step 102 can be arranged in pairs or groups mutually, form different embodiments.Such as, a kind of flow process of embodiment as shown in Figure 2, specifically comprises:
201, judged whether that data to be stored arrive, if judged result is yes, then performed step 202; If judged result is no, then perform step 205.
202, judge whether the data volume in local cache is greater than default first threshold; If judge that judged result is yes, then perform step 203; If judged result is no, then perform step 204.
203, by the data write into Databasce in local cache, empty local cache, by above-mentioned data write local cache to be stored after emptying local cache, and return execution step 201.
204, above-mentioned data to be stored are stored in local cache, and return execution step 201.
205, judge whether synchronizing cycle arrives, if judged result is yes, then perform step 206, if judged result is no, then returns and perform step 201.
206, judge whether have data in local cache, if judged result is yes, then perform step 207; If judged result is no, then returns and perform step 201.
207, by the data write into Databasce in local cache, and empty local cache, return and perform step 201.
The present embodiment can improve the handling capacity of data storage device, can ensure that again data are written into the real-time in database.
Further, in the various embodiments described above or embodiment, the situation that synchronizing cycle and data to be stored arrive simultaneously may be there is.For this situation, can pre-configured synchronous event priority and store Event Priority, the sequencing of operation corresponding to two kinds of conditions is performed in order to be limited to data storage device when above-mentioned two kinds of conditions arrive simultaneously.Specifically, when synchronizing cycle and data to be stored arrive simultaneously, data storage device obtains corresponding synchronous event priority and storage Event Priority corresponding to data to be stored synchronizing cycle, according to synchronous event priority with store Event Priority, determine data to be stored to be stored into the operation (i.e. above-mentioned steps 101) in local cache and/or database according to the first data storage logic and according to the second data storage logic, the data in local cache to be stored into the execution sequence of the operation (i.e. above-mentioned steps 102) in database.Afterwards, data storage device, according to determined execution sequence, performs and operates accordingly.Such as, if preset synchronous event priority higher than storage Event Priority, then, when synchronizing cycle and data syn-chronization to be stored arrive, data storage device preferentially performs the operation shown in step 102, then performs the operation shown in step 101.If preset on year-on-year basis, Event Priority is lower than storage Event Priority, then, when arriving when synchronizing cycle and data syn-chronization to be stored, data storage device preferentially performs the operation shown in step 101, then performs the operation shown in step 102.
Except aforesaid way, also can rob according to two kinds of events the sequencing accounting for resource, perform and operate accordingly.
It should be noted that, for aforesaid each embodiment of the method, in order to simple description, therefore it is all expressed as a series of combination of actions, but those skilled in the art should know, the present invention is not by the restriction of described sequence of movement, because according to the present invention, some step can adopt other orders or carry out simultaneously.Secondly, those skilled in the art also should know, the embodiment described in instructions all belongs to preferred embodiment, and involved action and module might not be that the present invention is necessary.
In the above-described embodiments, the description of each embodiment is all emphasized particularly on different fields, in certain embodiment, there is no the part described in detail, can see the associated description of other embodiments.
The structural representation of the data storage device that Fig. 3 provides for one embodiment of the invention.As shown in Figure 3, this equipment comprises: memory module 31 and synchronization module 32.
These data to be stored, for when there being data to be stored to arrive, are stored in local cache and/or database according to the first data storage logic by memory module 31.
Data in local cache, during for arriving in the synchronizing cycle of presetting, are stored in described database according to the second data storage logic by synchronization module 32.
Illustrate at this, the memory device of the present embodiment also comprises: local cache 33.Memory module 31 is connected with local cache 33 respectively with synchronization module 32.
In an Alternate embodiments, when memory module 31 has specifically been used in data to be stored arrival, judge whether the data volume in local cache is greater than default first threshold, if the data volume in local cache is not more than first threshold, then by data write local cache to be stored; If the data volume in local cache is greater than first threshold, then by the data write into Databasce in local cache, and by data write local cache to be stored after emptying local cache.Or
When memory module 31 has specifically been used in data to be stored arrival, by data write local cache to be stored, after by data write local cache to be stored, judge whether the data volume in local cache is greater than default first threshold, if the data volume in local cache is greater than first threshold, then by the data write into Databasce in local cache, and empty local cache; If the data volume in local cache is not more than first threshold, then terminate this time to store operation.Or
When memory module 31 has specifically been used in data to be stored arrival, judge whether the data volume in local cache is greater than default first threshold, if the data volume in local cache is not more than first threshold, then by data write local cache to be stored; If the data volume in local cache is greater than first threshold, then by the data in data to be stored and local cache in the lump write into Databasce, and empty local cache.
In an Alternate embodiments, when synchronization module 32 is specifically used in and arrives synchronizing cycle, judge whether have data in local cache, if having data in local cache, then by the data write into Databasce in local cache, and empty local cache.In addition, if do not have data in local cache, then synchronization module 32 can not perform any operation.Or
When synchronization module 32 is specifically used in and arrives synchronizing cycle, judge whether have data in local cache, if have data in local cache, then by the storage time in local cache of top n data write into Databasce the earliest, and N number of data of write into Databasce are deleted from local cache, N is the natural number of the data volume be less than in local cache.In addition, if do not have data in local cache, then synchronization module 32 can not perform any operation.Or
When synchronization module 32 is specifically used in and arrives synchronizing cycle, judge whether have data in local cache, if have data in local cache, then judge whether the data volume in local cache is greater than default Second Threshold, if the data volume in local cache is greater than Second Threshold, then by the data write into Databasce in local cache, and empty local cache.In addition, if there is no data in local cache, if or data volume in local cache be not more than Second Threshold, then synchronization module 32 can not perform any operation.
Preferably, Second Threshold is less than first threshold.
In an Alternate embodiments, as shown in Figure 4, data storage device also comprises: acquisition module 34 and determination module 35.
Acquisition module 34, for when arriving synchronizing cycle and data to be stored simultaneously, obtains corresponding synchronous event priority and storage Event Priority corresponding to data to be stored synchronizing cycle.
Determination module 35, be connected with acquisition module 34, for the synchronous event priority that obtains according to acquisition module 34 with store Event Priority, determine data to be stored to be stored into operation in local cache and/or database and the execution sequence according to the second data storage logic, the data in local cache being stored into the operation in database according to the first data storage logic.Wherein, memory module 31 is connected with determination module 35 respectively with synchronization module 32, and for when arriving synchronizing cycle and data to be stored simultaneously, the execution sequence determined according to determination module 35 performs corresponding operating.
What deserves to be explained is, the database that the present embodiment relates to can be any database, such as, can be Hbase.
Each functional module of the data storage device that the present embodiment provides can be used for the flow process performing embodiment of the method shown in Fig. 1 or Fig. 2, and its specific works principle repeats no more, and refers to the description of embodiment of the method.
The data storage device that the present embodiment provides, when there being data to be stored to arrive, according to the first data storage logic, data to be stored are stored in local cache and/or database, wherein, if first data to be stored are deposited local cache, be conducive to the handling capacity improving data storage device, if be directly stored in database by data to be stored, be then conducive to the real-time improving data storage; In addition, also synchronizing cycle is set, when arriving synchronizing cycle, according to the second data storage logic, the data in local cache are stored in database, can prevent data in local cache for a long time cannot the situation of write into Databasce, improve real-time data be stored in database to a certain extent.
Those skilled in the art can be well understood to, and for convenience and simplicity of description, the system of foregoing description, the specific works process of device and unit, with reference to the corresponding process in preceding method embodiment, can not repeat them here.
In several embodiment provided by the present invention, should be understood that, disclosed system, apparatus and method, can realize by another way.Such as, device embodiment described above is only schematic, such as, the division of described unit, be only a kind of logic function to divide, actual can have other dividing mode when realizing, such as multiple unit or assembly can in conjunction with or another system can be integrated into, or some features can be ignored, or do not perform.Another point, shown or discussed coupling each other or direct-coupling or communication connection can be by some interfaces, and the indirect coupling of device or unit or communication connection can be electrical, machinery or other form.
The described unit illustrated as separating component or can may not be and physically separates, and the parts as unit display can be or may not be physical location, namely can be positioned at a place, or also can be distributed in multiple network element.Some or all of unit wherein can be selected according to the actual needs to realize the object of the present embodiment scheme.
In addition, each functional unit in each embodiment of the present invention can be integrated in a processing unit, also can be that the independent physics of unit exists, also can two or more unit in a unit integrated.Above-mentioned integrated unit both can adopt the form of hardware to realize, and the form that hardware also can be adopted to add SFU software functional unit realizes.
The above-mentioned integrated unit realized with the form of SFU software functional unit, can be stored in a computer read/write memory medium.Above-mentioned SFU software functional unit is stored in a storage medium, comprising some instructions in order to make a computer equipment (can be personal computer, server, or the network equipment etc.) or processor (processor) perform the part steps of method described in each embodiment of the present invention.And aforesaid storage medium comprises: USB flash disk, portable hard drive, ROM (read-only memory) (Read-OnlyMemory, ROM), random access memory (RandomAccessMemory, RAM), magnetic disc or CD etc. various can be program code stored medium.
Last it is noted that above embodiment is only in order to illustrate technical scheme of the present invention, be not intended to limit; Although with reference to previous embodiment to invention has been detailed description, those of ordinary skill in the art is to be understood that: it still can be modified to the technical scheme described in foregoing embodiments, or carries out equivalent replacement to wherein portion of techniques feature; And these amendments or replacement, do not make the essence of appropriate technical solution depart from the spirit and scope of various embodiments of the present invention technical scheme.
Claims (10)
1. a date storage method, is characterized in that, comprising:
When there being data to be stored to arrive, according to the first data storage logic, described data to be stored are stored in local cache and/or database;
When arriving the synchronizing cycle of presetting, according to the second data storage logic, the data in described local cache are stored in described database.
2. method according to claim 1, is characterized in that, describedly described data to be stored is stored in local cache and/or database according to the first data storage logic, comprising:
Judge whether the data volume in described local cache is greater than default first threshold, if the data volume in described local cache is not more than described first threshold, then described data to be stored are write in described local cache; If the data volume in described local cache is greater than described first threshold, then the data in described local cache are write described database, and after emptying described local cache, described data to be stored are write described local cache; Or
Described data to be stored are write described local cache, after described data to be stored are write described local cache, judge whether the data volume in described local cache is greater than default first threshold, if the data volume in described local cache is greater than described first threshold, then the data in described local cache are write described database, and empty described local cache; If the data volume in described local cache is not more than described first threshold, then terminate this time to store operation; Or
Judge whether the data volume in described local cache is greater than default first threshold, if the data volume in described local cache is not more than described first threshold, then described data to be stored are write in described local cache; If the data volume in described local cache is greater than described first threshold, then the data in described data to be stored and described local cache is write described database in the lump, and empty described local cache.
3. method according to claim 1 and 2, is characterized in that, describedly data in described local cache is stored in described database according to the second data storage logic, comprising:
Judge whether have data in described local cache, if having data in described local cache, then the data in described local cache are write described database, and empty described local cache; Or
Judge whether have data in described local cache, if have data in described local cache, then the storage time in described local cache of top n data are the earliest write described database, and N number of data of database described in said write are deleted from described local cache, N is the natural number of the data volume be less than in described local cache; Or
Judge whether have data in described local cache, if have data in described local cache, then judge whether the data volume in described local cache is greater than default Second Threshold, if the data volume in local cache is greater than described Second Threshold, then the data in described local cache are write described database, and empty described local cache.
4. method according to claim 1 and 2, it is characterized in that, when described synchronizing cycle and described data to be stored arrive simultaneously, obtain corresponding synchronous event priority and storage Event Priority corresponding to described data to be stored described synchronizing cycle, according to described synchronous event priority and described storage Event Priority, determine described described data to be stored to be stored into operation in local cache and/or database and the described execution sequence according to the second data storage logic, the data in described local cache being stored into the operation in described database according to the first data storage logic.
5. method according to claim 1 and 2, is characterized in that, described database is Hbase.
6. a data storage device, is characterized in that, comprising:
Described data to be stored, for when there being data to be stored to arrive, are stored in local cache and/or database according to the first data storage logic by memory module;
Data in described local cache, during for arriving in the synchronizing cycle of presetting, are stored in described database according to the second data storage logic by synchronization module.
7. equipment according to claim 6, it is characterized in that, described memory module is specifically for when there being data to be stored to arrive, judge whether the data volume in described local cache is greater than default first threshold, if the data volume in described local cache is not more than described first threshold, then described data to be stored are write in described local cache; If the data volume in described local cache is greater than described first threshold, then the data in described local cache are write described database, and after emptying described local cache, described data to be stored are write in described local cache; Or
Described memory module is specifically for when there being data to be stored to arrive, described data to be stored are write described local cache, after described data to be stored are write described local cache, judge whether the data volume in described local cache is greater than default first threshold, if the data volume in described local cache is greater than described first threshold, then the data in described local cache are write described database, and empty described local cache; If the data volume in described local cache is not more than described first threshold, then terminate this time to store operation; Or
Described memory module is specifically for when there being data to be stored to arrive, judge whether the data volume in described local cache is greater than default first threshold, if the data volume in described local cache is not more than described first threshold, then described data to be stored are write in described local cache; If the data volume in described local cache is greater than described first threshold, then the data in described data to be stored and described local cache is write described database in the lump, and empty described local cache.
8. the equipment according to claim 6 or 7, it is characterized in that, described synchronization module is specifically for when arriving described synchronizing cycle, judge whether have data in described local cache, if have data in described local cache, then the data in described local cache are write described database, and empty described local cache; Or
Described synchronization module is specifically for when arriving described synchronizing cycle, judge whether have data in described local cache, if have data in described local cache, then the storage time in described local cache of top n data are the earliest write described database, and N number of data of database described in said write are deleted from described local cache, N is the natural number of the data volume be less than in described local cache; Or
Described synchronization module is specifically for when arriving described synchronizing cycle, judge whether have data in described local cache, if have data in described local cache, then judge whether the data volume in described local cache is greater than default Second Threshold, if the data volume in local cache is greater than described Second Threshold, then the data in described local cache are write described database, and empty described local cache.
9. the equipment according to claim 6 or 7, is characterized in that, also comprises:
Acquisition module, for when arriving described synchronizing cycle and described data to be stored simultaneously, obtains corresponding synchronous event priority and storage Event Priority corresponding to described data to be stored described synchronizing cycle;
Determination module, for according to described synchronous event priority and described storage Event Priority, determine describedly described data to be stored to be stored into operation in local cache and/or database and the described execution sequence according to the second data storage logic, the data in described local cache being stored into the operation in described database according to the first data storage logic.
10. the equipment according to claim 6 or 7, is characterized in that, described database is Hbase.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410333794.3A CN105446893A (en) | 2014-07-14 | 2014-07-14 | Data storage method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410333794.3A CN105446893A (en) | 2014-07-14 | 2014-07-14 | Data storage method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105446893A true CN105446893A (en) | 2016-03-30 |
Family
ID=55557119
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410333794.3A Pending CN105446893A (en) | 2014-07-14 | 2014-07-14 | Data storage method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105446893A (en) |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106202307A (en) * | 2016-07-01 | 2016-12-07 | 百势软件(北京)有限公司 | A kind of batch log preservation method and device |
| CN106354667A (en) * | 2016-08-30 | 2017-01-25 | 浪潮(北京)电子信息产业有限公司 | Method and device for solid-state drive cache control |
| CN106407307A (en) * | 2016-08-31 | 2017-02-15 | 天津南大通用数据技术股份有限公司 | Interaction method and interaction device for node data of distributed database |
| CN107229673A (en) * | 2017-04-20 | 2017-10-03 | 努比亚技术有限公司 | Method for writing data, Hbase terminals and the storage medium of Hbase databases |
| CN107491314A (en) * | 2017-08-30 | 2017-12-19 | 四川长虹电器股份有限公司 | Processing method is write based on Read-Write Locks algorithm is accessible to HBASE real time datas |
| CN107566434A (en) * | 2016-07-01 | 2018-01-09 | 北京京东尚科信息技术有限公司 | Data cached reading/writing method and system in a kind of distributed system |
| CN108563744A (en) * | 2018-04-12 | 2018-09-21 | 武汉斗鱼网络科技有限公司 | Slow querying method, device and terminal device based on Redis databases |
| CN108574870A (en) * | 2018-04-13 | 2018-09-25 | 郑州云海信息技术有限公司 | A kind of display methods, device and the equipment in multi channel signals source |
| CN108694187A (en) * | 2017-04-07 | 2018-10-23 | 北京国双科技有限公司 | The storage method and device of real-time streaming data |
| CN109413166A (en) * | 2018-10-09 | 2019-03-01 | 浙江明度智控科技有限公司 | A kind of industrial gateway and its data managing method |
| CN110990435A (en) * | 2019-12-03 | 2020-04-10 | 秒针信息技术有限公司 | Data synchronization method, device and computer readable storage medium |
| CN111090705A (en) * | 2018-10-23 | 2020-05-01 | 杭州海康威视数字技术股份有限公司 | Multidimensional data processing method, multidimensional data processing device, multidimensional data processing equipment and storage medium |
| CN112948440A (en) * | 2021-03-09 | 2021-06-11 | 北京小米移动软件有限公司 | Page data processing method and device, terminal and storage medium |
| CN114371810A (en) * | 2020-10-15 | 2022-04-19 | 中国移动通信集团设计院有限公司 | Data storage method and device of HDFS |
| CN115686383A (en) * | 2022-12-30 | 2023-02-03 | 摩尔线程智能科技(北京)有限责任公司 | Control method and controller of memory and chip system |
-
2014
- 2014-07-14 CN CN201410333794.3A patent/CN105446893A/en active Pending
Cited By (23)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107566434A (en) * | 2016-07-01 | 2018-01-09 | 北京京东尚科信息技术有限公司 | Data cached reading/writing method and system in a kind of distributed system |
| CN107566434B (en) * | 2016-07-01 | 2022-01-07 | 北京京东尚科信息技术有限公司 | Cache data reading and writing method and system, electronic equipment and computer readable medium |
| CN106202307B (en) * | 2016-07-01 | 2019-10-11 | 百势软件(北京)有限公司 | A kind of batch log preservation method and device |
| CN106202307A (en) * | 2016-07-01 | 2016-12-07 | 百势软件(北京)有限公司 | A kind of batch log preservation method and device |
| CN106354667B (en) * | 2016-08-30 | 2019-03-15 | 浪潮(北京)电子信息产业有限公司 | A kind of buffer control method and device of solid state hard disk |
| CN106354667A (en) * | 2016-08-30 | 2017-01-25 | 浪潮(北京)电子信息产业有限公司 | Method and device for solid-state drive cache control |
| CN106407307A (en) * | 2016-08-31 | 2017-02-15 | 天津南大通用数据技术股份有限公司 | Interaction method and interaction device for node data of distributed database |
| CN108694187A (en) * | 2017-04-07 | 2018-10-23 | 北京国双科技有限公司 | The storage method and device of real-time streaming data |
| CN107229673A (en) * | 2017-04-20 | 2017-10-03 | 努比亚技术有限公司 | Method for writing data, Hbase terminals and the storage medium of Hbase databases |
| CN107491314A (en) * | 2017-08-30 | 2017-12-19 | 四川长虹电器股份有限公司 | Processing method is write based on Read-Write Locks algorithm is accessible to HBASE real time datas |
| CN108563744B (en) * | 2018-04-12 | 2021-07-23 | 武汉斗鱼网络科技有限公司 | Slow query method and device based on Redis database and terminal equipment |
| CN108563744A (en) * | 2018-04-12 | 2018-09-21 | 武汉斗鱼网络科技有限公司 | Slow querying method, device and terminal device based on Redis databases |
| CN108574870A (en) * | 2018-04-13 | 2018-09-25 | 郑州云海信息技术有限公司 | A kind of display methods, device and the equipment in multi channel signals source |
| CN109413166A (en) * | 2018-10-09 | 2019-03-01 | 浙江明度智控科技有限公司 | A kind of industrial gateway and its data managing method |
| CN109413166B (en) * | 2018-10-09 | 2021-07-27 | 浙江明度智控科技有限公司 | Industrial gateway and data management method thereof |
| CN111090705A (en) * | 2018-10-23 | 2020-05-01 | 杭州海康威视数字技术股份有限公司 | Multidimensional data processing method, multidimensional data processing device, multidimensional data processing equipment and storage medium |
| CN111090705B (en) * | 2018-10-23 | 2023-08-25 | 杭州海康威视数字技术股份有限公司 | Multidimensional data processing method, device and equipment and storage medium |
| CN110990435A (en) * | 2019-12-03 | 2020-04-10 | 秒针信息技术有限公司 | Data synchronization method, device and computer readable storage medium |
| CN114371810A (en) * | 2020-10-15 | 2022-04-19 | 中国移动通信集团设计院有限公司 | Data storage method and device of HDFS |
| CN114371810B (en) * | 2020-10-15 | 2023-10-27 | 中国移动通信集团设计院有限公司 | Data storage method and device of HDFS |
| CN112948440A (en) * | 2021-03-09 | 2021-06-11 | 北京小米移动软件有限公司 | Page data processing method and device, terminal and storage medium |
| CN115686383A (en) * | 2022-12-30 | 2023-02-03 | 摩尔线程智能科技(北京)有限责任公司 | Control method and controller of memory and chip system |
| CN115686383B (en) * | 2022-12-30 | 2023-04-18 | 摩尔线程智能科技(北京)有限责任公司 | Control method and controller of memory and chip system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105446893A (en) | Data storage method and device | |
| CN105447075B (en) | The computer implemented method divided for dynamic | |
| CN105573660B (en) | Method and apparatus for improving the performance of sub-clustering disk array | |
| US8495313B2 (en) | Transferring learning metadata between storage servers having clusters via copy services operations on a shared virtual logical unit that stores the learning metadata | |
| CN106503020B (en) | Log data processing method and device | |
| CN109614377A (en) | File delet method, device, equipment and the storage medium of distributed file system | |
| CN103744906A (en) | System, method and device for data synchronization | |
| CN106844108B (en) | A kind of date storage method, server and storage system | |
| CN103049317B (en) | Based on the high concurrent data no write de-lay system and method for queue under cloud environment | |
| US20120137094A1 (en) | Snapshot based replication | |
| KR101574451B1 (en) | Imparting durability to a transactional memory system | |
| CN111124277A (en) | Deep learning data set caching method, system, terminal and storage medium | |
| CN103927338A (en) | Log information storage processing method and log information storage processing device | |
| CN103440204B (en) | A kind of method of updating file system and memory device | |
| CN104639347A (en) | Multi-cluster monitoring method and device, and system | |
| EP3051408A1 (en) | Data operating method and device | |
| CN106201561B (en) | The upgrade method and equipment of distributed caching cluster | |
| CN105824846A (en) | Data migration method and device | |
| CN109379305A (en) | A kind of data distributing method, device, server and storage medium | |
| CN104484131A (en) | Device and corresponding method for processing data of multi-disk servers | |
| CN109933461A (en) | A kind of method and apparatus of data processing | |
| CN103514140B (en) | For realizing the reconfigurable controller of configuration information multi-emitting in reconfigurable system | |
| CN105224418A (en) | A kind of data back up method and device | |
| CN104503868A (en) | Data synchronizing method, device and system | |
| CN104536852B (en) | Data recovery method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20160330 |