CN104572737B - Data storage householder method and system - Google Patents
Data storage householder method and system Download PDFInfo
- Publication number
- CN104572737B CN104572737B CN201310503949.9A CN201310503949A CN104572737B CN 104572737 B CN104572737 B CN 104572737B CN 201310503949 A CN201310503949 A CN 201310503949A CN 104572737 B CN104572737 B CN 104572737B
- Authority
- CN
- China
- Prior art keywords
- data
- ldb
- mdb
- thread
- modification
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
Abstract
This application discloses data storage householder method and system, wherein, data-storage system is using database D B, perdurable data storehouse LDB, memory database MDB tertiary structures, when application system sends inquiry request, inquired about first in MDB, if MDB is miss, the data that inquire are returned by inquiring about LDB, and by the write back data inquired in LDB to MDB；As LDB is also miss, then the data inquired are returned to by inquiring about DB, and by the write back data inquired in DB to LDB；Methods described includes：Judge whether the data in DB are changed；If, then corresponding data in LDB are modified, and corresponding data in MDB are subjected to invalidation, during to send the inquiry request to the data changed in application system, inquired about first in MDB, after finding that MDB is miss, then by inquiring about the data changed described in LDB returns.By the application, the read request processing pressure s of database can be reduced, improves the response efficiency to write operation requests.
Description
Technical field
The application is related to technical field of data storage, more particularly to data storage householder method and system.
Background technology
In some applications or service system, it is often necessary to some related data storages in database, and root
Handled accordingly according to the needs of management.In E-commerce transaction platform, seller user can be by business object（Such as business
Product, service etc.）Information, including inventory information etc., be published in transaction platform.In transaction platform, it is necessary to according to selling family expenses
The setting at family, the inventory information of conservation object in database, also, when buyer user produces buying behavior for certain object
When, it is necessary to reduced to the stockpile number of business object, other users then may be used when inquiring about the stockpile number of the business object
To show newest stockpile number.
But in the case of high concurrent, big request amount, the read-write requests amount of database can be increased suddenly, it is easy to reach
To the limit of database processing ability, it can so cause the partial function of system can not normal use.Therefore, database gradually into
For the fast-developing bottleneck of system for restricting.
Solution universal at present uses the two-level memory framework of caching (MDB)+database (DB) substantially, wherein, MDB
In internal memory, read or write speed is far longer than DB, and TPS when wall scroll record size is 512 byte reaches 70,000, response time RT<
2ms.Referring to Fig. 1, in MDB+DB two-level memory framework, when an inquiry operation is come temporarily, common way is：
1.0：Inquired about first in MDB, if hit in MDB, directly return to Query Result；
1.1：If miss in MDB, go to inquire about DB；
1.2：The result inquired in DB is written back in MDB.
When a write operation requests reach, common way is：
2.0：The data in DB are modified first；
2.1：Corresponding data in MDB are subjected to invalidation, such as carry out the operation such as deleting.
For example, it is assumed that producing an inquiry request in certain application, corresponding query object is data A, and now, inquiry request is first
MDB is first reached, if the record in MDB be present, is directly returned from MDB, otherwise, the inquiry request can be sent to DB, DB
After returning to Query Result, Query Result is written in MDB.So, during follow-up still other application query data A, then
Directly returned from MDB, without inquiring about DB again.
If follow-up have using the modification generated to data A again, data A can be modified in DB, and by MDB
In the record deletion on data A；Now, will in MDB if the inquiry to data A is initiated in still other application
It can not inquire, and be sent in DB and inquired about, Query Result is obtained in DB and then is written in MDB.So
The uniformity of data between MDB and DB can be ensured, and then ensure the correctness of Query Result.
It can be seen that this two-level memory structure reduces directly to DB inquiry request, advantageously reduce and apply when writing DB
Response time.But problems with least be present：According to the characteristics of MDB in itself, data are all stored in internal memory, and due to internal memory
Size limitation, data space can not any dilatation, this to store it is restricted, and easily due to a variety of causes（Such as
Server resets etc.）Data are lost, data integrity cannot be guaranteed.In addition, when write request is a lot, due to this method meeting
Corresponding data in MDB are emptied, therefore so that hit rate of the read request afterwards in MDB drastically declines, so as to DB
Protection substantially reduce, and then cause DB the response efficiency of write operation requests is also decreased.
The content of the invention
This application provides data storage householder method and system, database D B read request processing pressure can be reduced,
Improve the response efficiency to write operation requests.
This application provides following scheme：
A kind of data storage householder method, data-storage system is using database D B, perdurable data storehouse LDB, internal memory number
According to storehouse MDB tertiary structures, when application system sends inquiry request, inquire about in MDB, if MDB is miss, pass through first
Inquiry LDB returns to the data that inquire, and by the write back data inquired in LDB to MDB；As LDB is also miss, then by looking into
Ask DB and return to the data that inquire, and by the write back data inquired in DB to LDB；Methods described includes：
Judge whether the data in DB are changed；
If it is, being modified to corresponding data in LDB, and corresponding data in MDB are subjected to invalidation, with
Just when application system sends the inquiry request to the data changed, inquire about in MDB, do not ordered in discovery MDB first
In after, then by inquire about LDB return described in the data changed.
A kind of data storage accessory system, data-storage system is using database D B, perdurable data storehouse LDB, internal memory number
According to storehouse MDB tertiary structures, when application system sends inquiry request, inquire about in MDB, if MDB is miss, pass through first
Inquiry LDB returns to the data that inquire, and by the write back data inquired in LDB to MDB；As LDB is also miss, then by looking into
Ask DB and return to the data that inquire, and by the write back data inquired in DB to LDB；The accessory system includes：
Judging unit, for judging whether the data in DB are changed；
Modification and invalidation unit, if the judged result for the judging unit is yes, corresponding in LDB
Data are modified, and corresponding data in MDB are carried out into invalidation, have been changed to be sent in application system described
During the inquiry request of data, inquire about in MDB, after finding that MDB is miss, then repaiied by inquiring about described in LDB returns first
The data changed.
The specific embodiment provided according to the application, this application discloses following technique effect：
By the embodiment of the present application, data-storage system can use DB, LDB, MDB tertiary storage structure, and when hair
During raw write operation requests, it is ensured that the data consistency between LDB and DB, while again because LDB has the characteristics of persistence,
Therefore, in the case where read operation request can not be hit in MDB, can typically be hit in LDB layers, so as to avoid largely reading
Operation requests are sent to DB, are more beneficial for the protection to DB.Simultaneously as MDB, DB still have, therefore, can use
MDB reads and writes the advantages of speed highest, and can use the advantages of DB supports affairs and complex query, in a word, equivalent to
So that while the advantages of whole storage system is when retaining DB, MDB two-level memory, strengthen the protection to DB.And for DB and
Speech, it is interactional between read operation request and write operation requests, in the situation for the read operation request quantity reduction for reaching DB
Under, the DB write operation response time can also be reduced, therefore be advantageous to improve the performance of storage system on the whole.
Wherein, specifically when corresponding data are modified in LDB, can so may be used by the way of asynchronous modification
To avoid influenceing response times of the DB to write operation, and it is also beneficial to that the different modifications to same data are produced according to modification
Raw sequencing carries out serial process, in the case where a large amount of write operation requests are concurrent, also ensures that LDB is consistent with DB data
Property.
Certainly, any product for implementing the application it is not absolutely required to reach all the above advantage simultaneously.
Brief description of the drawings
, below will be to institute in embodiment in order to illustrate more clearly of the embodiment of the present application or technical scheme of the prior art
The accompanying drawing needed to use is briefly described, it should be apparent that, drawings in the following description are only some implementations of the application
Example, for those of ordinary skill in the art, on the premise of not paying creative work, can also be obtained according to these accompanying drawings
Obtain other accompanying drawings.
Fig. 1 is storage organization schematic diagram of the prior art；
Fig. 2 is the flow chart for the method that the embodiment of the present application provides；
Fig. 3 is the storage organization schematic diagram that the embodiment of the present application provides；
Fig. 4 is the schematic diagram for the system that the embodiment of the present application provides.
Embodiment
Below in conjunction with the accompanying drawing in the embodiment of the present application, the technical scheme in the embodiment of the present application is carried out clear, complete
Site preparation describes, it is clear that described embodiment is only some embodiments of the present application, rather than whole embodiments.It is based on
Embodiment in the application, the every other embodiment that those of ordinary skill in the art are obtained, belong to the application protection
Scope.
In the embodiment of the present application, in order to preferably being protected to DB, it is proposed that a kind of multistage storage organization,
That is, increasing by one layer of storage between the DB and MDB of prior art, this layer of storage is realized using LDB.Then guarantor is passed through
The uniformity of data between card LDB and DB so that miss MDB inquiry request, it is most of that inquiry knot can be obtained in LDB
Fruit, so as to reduce the inquiry request to DB, preferably play a part of protecting DB.
Wherein, LDB is a kind of perdurable data storehouse (also referred to as persistence caching), and it is read and write speed and is much larger than DB, according to statistics,
TPS when wall scroll record size is 512 byte can reach 50,000, response time RT<=2ms.It can be seen that its read-write efficiency will be less than
MDB, for DB, it does not support affairs, does not support complex query, and flexibility is inadequate, but for single read-write requests,
Read-write efficiency contrast DB has very big advantage, and data safety, is persistent storage.
Therefore, in the embodiment of the present application, used DB-LDB-MDB tertiary storage structures, directly replace MDB or
Person DB so that storage use can use the advantages of respective.Firstly, since MDB, in internal memory, it, which reads and writes speed, is
Highest, therefore retained；DB can support the inquiry of affairs and complexity, therefore also be retained.So, in the application
In the storage architecture of embodiment, for read operation request, request arrives first at MDB, if MDB is hit, is directly returned by MDB
The data inquired are returned, otherwise read operation request reaches LDB, if LDB is hit, the data inquired are returned by LDB, and
By the write back data inquired in LDB into MDB；If LDB is not also hit, read operation request can just be sent to DB, and
By the write back data inquired in DB to LDB.That is, the data in MDB, are the inquiry requests according to application, from LDB
Write-back, the data in LDB, be the inquiry request according to application, the write-back from DB.Certainly, in actual applications, in data
In the case that inquiry request is miss in MDB and LDB, the data that are inquired in DB can also be written back to simultaneously LDB with
And in MDB, so, the request for having the inquiry data next time again is that Query Result is directly obtained with MDB.
That is, in an initial condition, data are only stored in DB, with the generation of read operation request, it is read
Data can be progressively write in LDB and MDB；When producing write operation requests in application system, after the data in DB are changed,
Data in LDB are also changed, and corresponding data are then deactivated in MDB, and new read request can not get inquiry knot from MDB
Fruit, but Query Result can be got from LDB in most cases；When service is restarted, the data in MDB
It can lose, but the data in LDB can not lose.
In specific implementation, the embodiment of the present application it is also required to provide a system aided in, so as in DB, LDB, MDB tri-
Solves the problems, such as data consistency between level.Explanation is needed exist for, on the Data Consistency between this three-level, base
This starting point is so that the data between LDB and DB are consistent as far as possible, that is, when according to write operation requests in DB
After data are modified, corresponding data in LDB are also modified, and by corresponding data invalid in MDB.So,
When producing the request for inquiring about the data in application system, will not be hit in MDB, and next inquiry request can be sent
To LDB, because corresponding data have been carried out changing in LDB, therefore, it becomes possible to Query Result is provided from LDB layers, without
To be inquired about again into DB.Just the specific implementation of the accessory system is described in detail below.
Referring to Fig. 2, the embodiment of the present application provide firstly a kind of data storage householder method, and this method can include following
Step：
S201：Judge whether the data in DB are changed；
S202：If it is, being modified to corresponding data in LDB, and corresponding data in MDB are subjected to invalid place
Reason, during to send the inquiry request to the data changed in application system, inquired about first in MDB, finding MDB
After miss, then by inquiring about the data changed described in LDB returns.
Wherein, specifically modify in corresponding data in LDB and to MDB in corresponding data when carrying out invalid,
It can be realized by interface corresponding to calling.
During specific implementation, modification can be synchronized to the data in LDB.That is, storage system is receiving one
After the write operation requests that individual application system is sent, the data in DB can be modified in a thread（Including data
Renewal, deletion, newly-built etc.）, in the case of synchronous vacations LDB, can directly in the thread to LDB in corresponding data
Modify, returned again to after the data modification success in LDB.In the case where the concurrent quantity of write operation requests is little, this
The mode of kind synchronous vacations can more timely realize the data consistency in LDB and DB.But in the concurrent quantity of write request very
In the case of big, then phenomena such as implementation of concurrent modification LDB data may result in error.For example, in the feelings of high concurrent
Under condition, the multiple modification on same data may be produced simultaneously, now, if synchronous right in the thread of modification DB data
Data in LDB are modified, then the response time of possible error or extension system.Such as, it is assumed that the data changed in DB
The stockpile number of certain merchandise items, it is assumed that the quantity before modification is 100, certain moment receive first write request, it is necessary to
Reduce 1, now, the data in DB are modified to 99, during data in synchronous vacations LDB, also by corresponding stockpile number by
100 are revised as 99；But just after first write request is received, a write request is received again, ask again by the commodity pair
The stockpile number of elephant reduces 2, now, may not yet be returned for the thread of first write request, it is necessary to wait first to write
After the thread of request returns, second write request could be handled, it is otherwise, directly right if being not to wait for first thread and returning
Data modification in DB and LDB, it will malfunction.Therefore, in the embodiment of the present application, asynchronous modification LDB numbers can also be used
According to implementation.
That is, the thread for changing data in DB, can returns after the data during DB is completed in modification, and
On the modification of corresponding data in LDB, then can be carried out in another thread.For ease of describing, can be other by this
Thread is referred to as " asynchronous task processing thread ".That is, the data modification in data modification and DB in LDB is not complete
Synchronously carry out, but the data modification in DB is completed and then handles thread to corresponding data in LDB by asynchronous task
Modify.So, the thread that can to change DB data more timely returns, and improves the speed of response of write request.Certainly,
When asynchronous task processing thread is modified to the data in LDB, can be carried out according to the order that data in DB are changed.
During specific implementation, corresponding data in LDB are changed in order to the order changed according to data in DB, can be with
Monitor DB journal file（Such as Binlog daily records of Mysql databases etc.）Change, it is for instance possible to use timing check
Mode is monitored；When finding to produce new record in journal file, it can extract and be parsed（In journal file
Every record in preserve the information such as action type）, by analysis result it may determine that whether the data gone out in DB occur
Modification, also, time caused by each bar record in journal file, reflect the sequencing that the data in DB are changed, because
This, in asynchronous task handles thread, it is possible to the sequencing being created in journal file is recorded according to each bar, to LDB
In corresponding data modify, and corresponding data in MDB are subjected to invalidation.
In actual applications, in order to improve the ability of reply high concurrent situation, can be created in storage system multiple asynchronous
Task handles thread, and the operation for changing LDB corresponding datas can be assigned to different asynchronous tasks and handle thread by accessory system
Upper execution, be so advantageous to improve treatment effeciency, reduce from DB data be modified to LDB data modifications completion between prolong
When.Certainly, for different modification records caused by same data, then can be assigned on same asynchronous task processing thread
Perform, can so enable asynchronous task processing thread to be performed according to the different time sequencings that are created of modification record pair
The modification operation of corresponding data, avoids malfunctioning in LDB.
During specific implementation, because each data in DB generally have unique identification, for example, for e-commerce transaction
For merchandise items this data in platform, typically unique identification, this ItemID mono- can be carried out respectively with ItemID
As be digital ID, namely represented with string number；In addition, each asynchronous task processing thread can also have respective number
Word is numbered, it may for example comprise asynchronous task processing thread of numbering 0,1,2 ..., etc..Therefore, it is same in order to be directed to
The different modification records of one data are assigned on same asynchronous task processing thread and handled, and it is total can to handle thread with asynchronous task
Number is modulus, and the unique identification of the data to being changed carries out modulus computing, by asynchronous that numbering is modulus operation result
Business processing thread handles thread as the target asynchronous task.For example, repaiied when extracting the first data from journal file
When changing record, the ID for the data changed can be therefrom got first, for example, 0002, it is assumed that asynchronous task processing thread is total
Number is 100, then after carrying out modulus computing, obtained result is 2, accordingly, it is possible to which data modification record is distributed into numbering
Thread is handled for 2 asynchronous task；Next, if the ID for the data that the record display of Article 2 data modification is changed is also
0002, then data modification record, which can be also assigned to, numbers the asynchronous task for being 2 processing thread, by that analogy.Thus
It can ensure all to be assigned to same asynchronous task processing thread for the different modification records of same data, at the asynchronous task
Thread is managed can then to perform serial modification according to the generation sequencing of different modification records to corresponding data in LDB, keep away
Exempt from mistake occur, cause the data in the data and DB in LDB inconsistent.
Certainly, it is necessary to illustrate, because in the embodiment of the present application, data repaiies in the modification and DB to data in LDB
It is asynchronous progress to change, and therefore, data modification is accomplished to data modification in LDB and completes typically produce certain prolong from DB
When, this delay is usually that Millisecond is other, and therefore, typically specific operation system will not be had an impact.
It is further to note that in the embodiment of the present application, it is complete to be accomplished to data modification in LDB for data modification from DB
Delay between, it is related with the asynchronous task processing number of threads created in system, in the processing of each asynchronous task
The task amount of thread is all than in the case of more saturated, may be such that this delay increases.Therefore, when can pre-set one
Between threshold value, and in accessory system, each bar can also be recorded and asynchronous modification in LDB is created to from journal file completes
Between time interval counted, if time interval exceedes the preset time threshold, warning information can be sent, this
Sample, it is possible to further decided whether to carry out dilatation processing to asynchronous task processing thread according to the warning information, it is so-called
Dilatation processing, that is, handle according to asynchronous task the speed of thread process journal file, and recorded in journal file
Increment etc., creates more asynchronous task processing threads, and prolonging completing is produced to asynchronous modification LDB to reduce from daily record
When.
In order to more fully understand the technical scheme of the embodiment of the present application proposition, below by the storage in the embodiment of the present application
Structural representation and corresponding reading and writing data timing diagram, the embodiment of the present application is described in detail.
Referring to Fig. 3, it is the storage organization schematic diagram in the embodiment of the present application, it is seen that whole storage system be divided into DB,
LDB, MDB tertiary structure, wherein, MDB is in the superiors, is still within internal memory, and LDB is in intermediate layer.When application system is produced
During raw read operation request, specific timing diagram can be as shown in Fig. 3 1.0~1.4.Wherein：
1.0：When receiving read operation request, MDB is inquired about first；
1.1：If MDB is hit, Query Result directly is returned to application system, if do not hit in MDB, is inquired about
LDB；
1.2；If LDB is hit, Query Result is returned to application system, and Query Result is written back to MDB；If
LDB is miss, then inquires about DB；
1.3：The Query Result obtained in DB is written back to LDB；
1.4：The Query Result obtained in DB is written back to MDB.
And when application system produces write operation requests, referring to the step 2.0 in Fig. 3, that is, directly performing in modification DB
Data operation, DB modification complete after, corresponding thread i.e. can return to.
Then, in asynchronous task handles thread, 2.1 in Fig. 3 are performed：Enter line asynchronous to corresponding data in LDB to repair
Change, and 2.2：Invalidation is carried out to corresponding data in MDB.
In a word, in the embodiment of the present application, data-storage system can use DB, LDB, MDB tertiary storage structure, and
When write operation requests occur, it is ensured that the data consistency between LDB and DB, while again because LDB has persistence
Feature, therefore, in the case where read operation request can not be hit in MDB, it can typically be hit in LDB layers, it is big so as to avoid
The read operation request of amount is sent to DB, is more beneficial for the protection to DB., therefore, can be with simultaneously as MDB, DB still have
The advantages of using MDB read-write speed highests, and the advantages of DB supports affairs and complex query can be used, in a word,
While equivalent to the advantages of whole storage system is when retaining DB, MDB two-level memory is caused, strengthen the protection to DB.It is and right
It is interactional for DB, between read operation request and write operation requests, is reduced in the read operation request quantity for reaching DB
In the case of, the DB write operation response time can also be reduced, therefore be advantageous to improve the performance of storage system on the whole.
Corresponding with the data storage householder method that the embodiment of the present application provides, the embodiment of the present application additionally provides a kind of number
According to storage accessory system, likewise, specific data-storage system is using database D B, perdurable data storehouse LDB, internal storage data
Storehouse MDB tertiary structures, when application system sends inquiry request, inquired about first in MDB, if MDB is miss, by looking into
Ask LDB and return to the data that inquire, and by the write back data inquired in LDB to MDB；As LDB is also miss, then pass through inquiry
DB returns to the data that inquire, and by the write back data inquired in DB to LDB；Referring to Fig. 4, the accessory system can include with
Lower unit：
Judging unit 401, for judging whether the data in DB are changed；
Modification and invalidation unit 402, if the judged result for the judging unit is yes, to right in LDB
The data answered are modified, and corresponding data in MDB are carried out into invalidation, have been repaiied to be sent in application system to described
During the inquiry request of the data changed, inquired about first in MDB, after finding that MDB is miss, then by inquiring about described in LDB returns
The data changed.
Wherein, during specific implementation, modification and invalidation unit 402 can include：
Synchronization process subelement, in the thread for changing the data in DB, being carried out to corresponding data in LDB
Synchronous vacations, and corresponding data in MDB are subjected to invalidation.
Or under another implementation, for changing the thread of the data in DB the data modification in DB is complete
Into returning afterwards, the modification and invalidation unit 402 can include：
Asynchronous process subelement, used in handling thread in asynchronous task, line asynchronous is entered to corresponding data in LDB and repaiied
Change, and corresponding data in MDB are subjected to invalidation.
In the case of asynchronous process, judging unit 401 can specifically include：
Journal file monitors subelement, the change of the journal file for monitoring DB；
Parsing subelement is read, for when producing new record in the journal file, being read from the journal file
Take wherein caused record and parsed；
Judge subelement, whether the data for judging according to analysis result in DB are changed.
, can be according to each bar in the case where corresponding data are modified during the record in journal file is to LDB
The sequencing being recorded in journal file is handled, and accordingly, asynchronous process subelement specifically can be used for：
In asynchronous task handles thread, the sequencing being created in journal file is recorded according to each bar, in LDB
Corresponding data are modified, and corresponding data in MDB are carried out into invalidation.
Wherein, in order to improve the speed of asynchronous modification, the asynchronous task processing thread in system may be it is multiple, now,
Thread process, the asynchronous process subelement are handled by different asynchronous tasks in order to avoid the different modification records of same data
Specifically it can be used for：
According to the unique identification for the data changed in DB, it is determined that asynchronous for the target for changing corresponding data in LDB
Task handles thread, and thread is handled so as to which the different modification tasks for same data are sent into same target asynchronous task,
And in same target asynchronous task handles thread, according to time order and function order caused by modification task, to corresponding number in LDB
According to modifying.
During specific implementation, the unique identification is is represented by the ID of modification data, the ID with numeral, described asynchronous
Business processing thread has respective digital number, and the asynchronous process subelement can specifically include：
Modulus computing subelement, for handling total number of threads as modulus using asynchronous task, to the unique of the data changed
Property mark carry out modulus computing, using number be modulus operation result asynchronous task handle thread as the target asynchronous task
Handle thread.
In addition, the accessory system can also include：
Time interval computing unit, it is accomplished to corresponding data in LDB for being changed the data in DB and is repaiied by asynchronous
The time interval for changing completion is calculated；
Alarm Unit, for when the time interval exceedes preset time threshold, warning information being sent, so as to basis
The warning information carries out dilatation processing to asynchronous task processing thread.
When application system sends inquiry request, if MDB and LDB are miss, inquired by inquiring about DB returns
Data, and by the write back data inquired in DB to LDB and MDB.
In a word, in the embodiment of the present application, data-storage system can use DB, LDB, MDB tertiary storage structure, and
When write operation requests occur, it is ensured that the data consistency between LDB and DB, while again because LDB has persistence
Feature, therefore, in the case where read operation request can not be hit in MDB, it can typically be hit in LDB layers, it is big so as to avoid
The read operation request of amount is sent to DB, is more beneficial for the protection to DB., therefore, can be with simultaneously as MDB, DB still have
The advantages of using MDB read-write speed highests, and the advantages of DB supports affairs and complex query can be used, equivalent to
So that while the advantages of whole storage system is when retaining DB, MDB two-level memory, strengthen the protection to DB.And for DB and
Speech, it is interactional between read operation request and write operation requests, in the situation for the read operation request quantity reduction for reaching DB
Under, the DB write operation response time can also be reduced, therefore be advantageous to improve the performance of storage system on the whole.
As seen through the above description of the embodiments, those skilled in the art can be understood that the application can
Realized by the mode of software plus required general hardware platform.Based on such understanding, the technical scheme essence of the application
On the part that is contributed in other words to prior art can be embodied in the form of software product, the computer software product
It can be stored in storage medium, such as ROM/RAM, magnetic disc, CD, including some instructions are causing a computer equipment
（Can be personal computer, server, or network equipment etc.）Perform some of each embodiment of the application or embodiment
Method described in part.
Each embodiment in this specification is described by the way of progressive, identical similar portion between each embodiment
Divide mutually referring to what each embodiment stressed is the difference with other embodiment.Especially for system or
For system embodiment, because it is substantially similar to embodiment of the method, so describing fairly simple, related part is referring to method
The part explanation of embodiment.System and system embodiment described above is only schematical, wherein the conduct
The unit that separating component illustrates can be or may not be it is physically separate, can be as the part that unit is shown or
Person may not be physical location, you can with positioned at a place, or can also be distributed on multiple NEs.Can root
Factually border needs to select some or all of module therein realize the purpose of this embodiment scheme.Ordinary skill
Personnel are without creative efforts, you can to understand and implement.
Above to storage system householder method provided herein and system, it is described in detail, it is used herein
Specific case is set forth to the principle and embodiment of the application, and the explanation of above example is only intended to help and understands
The present processes and its core concept；Meanwhile for those of ordinary skill in the art, according to the thought of the application, having
There will be changes in body embodiment and application.In summary, this specification content should not be construed as to the application
Limitation.
Claims (10)
1. a kind of data storage householder method, it is characterised in that data-storage system is using database D B, perdurable data storehouse
LDB, memory database MDB tertiary structures, when application system sends inquiry request, are inquired about in MDB first, if MDB is not
Hit, then return to the data that inquire by inquiring about LDB, and by the write back data inquired in LDB to MDB；As LDB is not also ordered
In, then return to the data that inquire by inquiring about DB, and by the write back data inquired in DB to LDB；Methods described includes：
Judge whether the data in DB are changed；
If it is, modified to corresponding data in LDB, and corresponding data in MDB are subjected to invalidation, so as to
When application system sends the inquiry request to the data changed, inquire about in MDB, after finding that MDB is miss, then lead to first
Cross the data changed described in inquiry LDB returns.
2. according to the method for claim 1, it is characterised in that it is described that corresponding data in LDB are modified, and will
Corresponding data carry out invalidation in MDB, including：
In the thread for changing the data in DB, corresponding data in LDB are synchronized with modification, and will be corresponding in MDB
Data carry out invalidation.
3. according to the method for claim 1, it is characterised in that for changing the threads of the data in DB by the number in DB
Returned after being completed according to modification, it is described that corresponding data in LDB are modified, and corresponding data in MDB are subjected to nothing
Effect processing, including：
In asynchronous task handles thread, corresponding data in LDB are entered with line asynchronous modification, and corresponding data in MDB are entered
Row invalidation.
4. according to the method for claim 3, it is characterised in that whether the data judged in DB are changed, including：
Monitor the change of DB journal file；
When producing new record in the journal file, wherein caused new record is read from the journal file and is gone forward side by side
Row parsing；
Judge whether the data in DB are changed according to analysis result.
5. according to the method for claim 4, it is characterised in that it is described in asynchronous task handles thread, to corresponding in LDB
Data enter line asynchronous modification, and corresponding data in MDB are subjected to invalidation, including：
In asynchronous task handles thread, the sequencing being created in journal file is recorded according to each bar, to corresponding in LDB
Data modify, and corresponding data in MDB are subjected to invalidation.
6. according to the method for claim 3, it is characterised in that described when asynchronous task processing thread is multiple
In asynchronous task handles thread, corresponding data in LDB are entered with line asynchronous modification, and corresponding data in MDB are subjected to nothing
Effect processing, including：
According to the unique identification for the data changed in DB, it is determined that the target asynchronous task for changing corresponding data in LDB
Thread is handled, thread is handled so as to which the different modification tasks for same data are sent into same target asynchronous task, and
In same target asynchronous task processing thread, according to time order and function order caused by modification task, corresponding data in LDB are entered
Row modification.
7. according to the method for claim 6, it is characterised in that the unique identification be by the ID of modification data, it is described
ID represents with numeral, and the asynchronous task processing thread has a respective digital number, it is described according to the data changed only
One property identifies, it is determined that thread is handled for changing the target asynchronous task of corresponding data in LDB, including：
Total number of threads is handled as modulus using asynchronous task, the unique identification of the data to being changed carries out modulus computing, will compile
Number for modulus operation result asynchronous task handle thread as the target asynchronous task handle thread.
8. according to the method for claim 3, it is characterised in that also include：
Data in DB are changed and is accomplished to corresponding data in LDB and is calculated by the time interval that asynchronous modification is completed；
When the time interval exceedes preset time threshold, send warning information, so as to according to the warning information to different
Step task processing thread carries out dilatation processing.
9. according to the method described in any one of claim 1 to 8, it is characterised in that when application system sends inquiry request, such as
Fruit MDB and LDB is miss, then returns to the data that inquire by inquiring about DB, and by the write back data inquired in DB to LDB
And MDB.
10. a kind of data storage accessory system, it is characterised in that data-storage system is using database D B, perdurable data storehouse
LDB, memory database MDB tertiary structures, when application system sends inquiry request, are inquired about in MDB first, if MDB is not
Hit, then return to the data that inquire by inquiring about LDB, and by the write back data inquired in LDB to MDB；As LDB is not also ordered
In, then return to the data that inquire by inquiring about DB, and by the write back data inquired in DB to LDB；The accessory system bag
Include：
Judging unit, for judging whether the data in DB are changed；
Modification and invalidation unit, for if it is, modified to corresponding data in LDB, and will be in MDB corresponding to
Data carry out invalidation, during to send the inquiry request to the data changed in application system, are looked into first in MDB
Ask, after finding that MDB is miss, then by inquiring about the data changed described in LDB returns.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310503949.9A CN104572737B (en) | 2013-10-23 | 2013-10-23 | Data storage householder method and system |
| HK15106191.7A HK1205801A1 (en) | 2013-10-23 | 2015-06-30 | Auxiliary method for storing data and system thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310503949.9A CN104572737B (en) | 2013-10-23 | 2013-10-23 | Data storage householder method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104572737A CN104572737A (en) | 2015-04-29 |
| CN104572737B true CN104572737B (en) | 2018-01-30 |
Family
ID=53088821
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310503949.9A Active CN104572737B (en) | 2013-10-23 | 2013-10-23 | Data storage householder method and system |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN104572737B (en) |
| HK (1) | HK1205801A1 (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107783728B (en) * | 2016-08-31 | 2021-07-23 | 百度在线网络技术（北京）有限公司 | Data storage method, device and equipment |
| CN108959284B (en) * | 2017-05-17 | 2020-12-08 | 中国移动通信集团重庆有限公司 | Method and device for processing data in columnar database |
| CN110765076B (en) * | 2019-10-25 | 2023-04-21 | 北京奇艺世纪科技有限公司 | Data storage method, device, electronic equipment and storage medium |
| CN112632117A (en) * | 2020-12-30 | 2021-04-09 | 广州华多网络科技有限公司 | Method and device for processing serial number data, electronic equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6701359B1 (en) * | 1998-07-28 | 2004-03-02 | International Business Machines Corp. | Apparatus and method for managing a multi-threaded persistent agent management information base including a managed object instance cache |
| US6941295B2 (en) * | 1999-08-12 | 2005-09-06 | International Business Machines Corporation | Data access system |
| CN101510209A (en) * | 2009-03-30 | 2009-08-19 | 北京金山软件有限公司 | Method, system and server for implementing real time search |
| CN102542034A (en) * | 2011-12-23 | 2012-07-04 | 北京人大金仓信息技术股份有限公司 | Database interface-based result set cache method |
| CN102880557A (en) * | 2012-09-13 | 2013-01-16 | 浙江金大科技有限公司 | Multistage distribution type high-speed cache of heterogeneous data source |
-
2013
- 2013-10-23 CN CN201310503949.9A patent/CN104572737B/en active Active
-
2015
- 2015-06-30 HK HK15106191.7A patent/HK1205801A1/en unknown
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6701359B1 (en) * | 1998-07-28 | 2004-03-02 | International Business Machines Corp. | Apparatus and method for managing a multi-threaded persistent agent management information base including a managed object instance cache |
| US6941295B2 (en) * | 1999-08-12 | 2005-09-06 | International Business Machines Corporation | Data access system |
| CN101510209A (en) * | 2009-03-30 | 2009-08-19 | 北京金山软件有限公司 | Method, system and server for implementing real time search |
| CN102542034A (en) * | 2011-12-23 | 2012-07-04 | 北京人大金仓信息技术股份有限公司 | Database interface-based result set cache method |
| CN102880557A (en) * | 2012-09-13 | 2013-01-16 | 浙江金大科技有限公司 | Multistage distribution type high-speed cache of heterogeneous data source |
Non-Patent Citations (2)
| Title |
|---|
| "Redis在高速缓存系统中的应用";曾超宇 等;《微型机与应用》;20130630;第32卷;全文 * |
| "内存数据库在高速缓存方面的应用";杨艳 等;《现代电信科技》;20111231(第12期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104572737A (en) | 2015-04-29 |
| HK1205801A1 (en) | 2015-12-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10846277B1 (en) | Journaled tables in database systems | |
| CN105210062B (en) | System scope checkpoint for distributed data base system avoids | |
| CN104598459B (en) | database processing, data access method and system | |
| US11249980B2 (en) | Updating of in-memory synopsis metadata for inserts in database table | |
| CN104572737B (en) | Data storage householder method and system | |
| EP1808779B1 (en) | Bundling database | |
| CN101566986A (en) | Method and device for processing data in online business processing | |
| CN102521330A (en) | Mirror distributed storage method under desktop virtual environment | |
| US9116627B2 (en) | Variable page sizing for improved physical clustering | |
| CN109240946A (en) | The multi-level buffer method and terminal device of data | |
| CN110188108A (en) | Date storage method, device, system, computer equipment and storage medium | |
| CN109871181A (en) | A kind of Object Access method and device | |
| US10747773B2 (en) | Database management system, computer, and database management method | |
| CN102073690B (en) | Method for constructing memory database supporting historical Key information | |
| CN103502925A (en) | Management method and device of monitoring records | |
| CN109446258A (en) | A kind of distributed data storage method and system | |
| CN116339643B (en) | Formatting method, formatting device, formatting equipment and formatting medium for disk array | |
| CN107295059A (en) | The statistical system and method for service propelling amount | |
| US6741970B2 (en) | Method and apparatus for enhanced, high speed updating and storing of E-commerce orders in a server | |
| CN106202271A (en) | The read method of the product database of OTA | |
| CN106940660A (en) | The method and apparatus of the realization of caching | |
| US20080168025A1 (en) | Methods, systems, and computer program products for reducing database workload volume | |
| CN111435331A (en) | Data writing method and device for storage volume, electronic equipment and machine-readable storage medium | |
| US8615632B2 (en) | Co-storage of data storage page linkage, size, and mapping | |
| CN106527959A (en) | Processing method and device for refreshing disk input output request |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| REG | Reference to a national code |
Ref country code: HK Ref legal event code: DE Ref document number: 1205801 Country of ref document: HK |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| REG | Reference to a national code |
Ref country code: HK Ref legal event code: GR Ref document number: 1205801 Country of ref document: HK |