CN109981774A - Data cache method and data buffer storage - Google Patents
Data cache method and data buffer storage Download PDFInfo
- Publication number
- CN109981774A CN109981774A CN201910226551.2A CN201910226551A CN109981774A CN 109981774 A CN109981774 A CN 109981774A CN 201910226551 A CN201910226551 A CN 201910226551A CN 109981774 A CN109981774 A CN 109981774A
- Authority
- CN
- China
- Prior art keywords
- data
- cached
- disk areas
- memory
- content
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
- G06F12/0806—Multiuser, multiprocessor or multiprocessing cache systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
- G06F12/0844—Multiple simultaneous or quasi-simultaneous cache accessing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Memory System Of A Hierarchy Structure (AREA)
Abstract
Present disclose provides a kind of data cache methods.The described method includes: every data during acquisition first is data cached is cached to the storage corresponding relationship of memory from the first disk areas, wherein the described first data cached caching for the data being stored in first disk areas at the first moment；Changed one or more data of data content, the first period are during being continued until the current time after first moment in first disk areas within first period for acquisition；Based on the storage corresponding relationship, the cache information of one or more data in the memory is determined；And the cache information based on one or more data, one or more data are cached again, it is data cached to update described first.The disclosure additionally provides a kind of data buffer storage device.
Description
Technical field
This disclosure relates to a kind of data cache method and data buffer storage.
Background technique
Existing computing engines, such as Spark are usually to carry out when data cached every time to all data of a business
Whole caching.However, data are often dynamic change in practical business, such as business datum of every daily increment, and often
It is not that total data can all be synchronized and be changed.However, computing engines, which cannot achieve, is retaining the part caching number not changed
According to while, to changed partial data carry out buffer update.The prior art passes through all data weights to a business
New whole caching is data cached to update, this will lead to the data cached originally and all can be capped or abandon.And with number
According to measuring, increasing and data dynamic change is more and more frequent, if cached again to all data, meeting every time
Very big cluster is caused to bear, traffic affecting is normally carried out.
Summary of the invention
An aspect of this disclosure provides a kind of data cache method.The data cache method includes: to obtain first to delay
Every data in deposit data is cached to the storage corresponding relationship of memory from the first disk areas, wherein described first is data cached
For the caching to the data being stored at the first moment in first disk areas；Obtain first magnetic within first period
Changed one or more data of data content in disk area, the first period be after first moment always
During continueing to the current time；Based on the storage corresponding relationship, one or more data are determined in the memory
Cache information；And the cache information based on one or more data, one or more data are cached again,
It is data cached to update described first.
Optionally, it is described acquisition within first period in first disk areas data content it is changed one or
A plurality of data including the data query operation executed in the monitoring first period to first disk areas, and are based on
The data query operation obtains one or more data.
Optionally, it is described acquisition within first period in first disk areas data content it is changed one or
A plurality of data are recorded in the record log of the operation executed in the first period to first disk areas including obtaining,
The record log is analyzed to obtain and make the changed operation information of data content, and is based on the operation information acquisition institute
State one or more data.
Optionally, it is described acquisition within first period in first disk areas data content it is changed one or
A plurality of data, including recording the newest content alteration operation executed to every data in first disk areas in real time
Temporal information, and be based on the temporal information, obtain the time of newest content alteration operation in the first period
Interior total data, to obtain one or more data.
Optionally, the cache information based on one or more data, again by one or more data
Caching includes: the cache information based on one or more data, determines one or more data in the memory
Affiliated data subregion or deblocking；And it is first disk areas is currently stored with the data subregion or data
The corresponding data of piecemeal integrally cache again.
Optionally, the cache information based on one or more data, again by one or more data
Caching, including caching each data in one or more data again respectively according to corresponding cache information.
Optionally, each data by one or more data weighs respectively according to corresponding cache information
New caching includes: when one or more data include the content update data of legacy data in first disk areas
When, the legacy data cached in the memory is substituted with the content update data；When one or more data
When including newly-increased data newly-increased in first disk areas, increase the caching of the newly-increased data in the memory；
And when one or more data include deleting the initial data of content from first disk areas, described in deletion
The initial data cached in memory.
Another aspect of the present disclosure provides a kind of data buffer storage device.The data buffer storage device includes that corresponding relationship obtains
Modulus block, data in magnetic disk variation obtain module, caching change determining module and buffer update module.Corresponding relationship obtains mould
Block be used to obtain first it is data cached in every data the storage corresponding relationship of memory is cached to from the first disk areas, wherein
The described first data cached caching for the data being stored at the first moment in first disk areas.Data in magnetic disk becomes
Change acquisition module and is used to obtain within first period data content changed one or more in first disk areas
Data, the first period are during being continued until the current time after first moment.Caching, which changes, determines mould
Block is used to be based on the storage corresponding relationship, determines the cache information of one or more data.Buffer update module is used for
Based on the cache information of one or more data, one or more data are cached again, to update described first
It is data cached.
Optionally, the data in magnetic disk variation obtains module and is specifically used for: monitoring in the first period to described first
The data query operation that disk areas executes；And inquiry operation based on the data, obtain one or more data.
Optionally, the data in magnetic disk variation obtains module and is specifically used for: acquisition is recorded in the first period to institute
State the record log of the operation of the first disk areas execution；The record log is analyzed, acquisition keeps data content changed
Operation information；And it is based on the operation information, obtain one or more data.
Optionally, the data in magnetic disk variation obtains module and is specifically used for: record is in first disk areas in real time
Every data execute a newest content alteration operation temporal information；And it is based on the temporal information, it obtains newest
Total data of the time of content alteration operation within the first period, to obtain one or more data.
Optionally, the buffer update module is specifically used for: the cache information based on one or more data, determines
One or more data in the memory belonging to data subregion or deblocking, and by first disk areas
Currently stored data corresponding with the data subregion or deblocking integrally cache again.
Optionally, the buffer update module be specifically used for by each data in one or more data according to
Corresponding cache information caches again respectively.
Optionally, each data by one or more data weighs respectively according to corresponding cache information
New caching includes: when one or more data include the content update data of legacy data in first disk areas
When, the legacy data cached in the memory is substituted with the content update data；When one or more data
When including newly-increased data newly-increased in first disk areas, increase the caching of the newly-increased data in the memory；
And when one or more data include deleting the initial data of content from first disk areas, described in deletion
The initial data cached in memory.
Another aspect of the present disclosure provides a kind of data buffering system.The data buffering system includes storage computer
The one or more memories and one or more processors of executable instruction.The processor executes described instruction with reality
Now data cache method as described above.
Another aspect of the present disclosure provides a kind of computer readable storage medium, is stored with computer executable instructions,
Described instruction is when executed for realizing data cache method as described above.
Another aspect of the present disclosure provides a kind of computer program, and the computer program, which includes that computer is executable, to be referred to
It enables, described instruction is when executed for realizing data cache method as described above.
Detailed description of the invention
In order to which the disclosure and its advantage is more fully understood, referring now to being described below in conjunction with attached drawing, in which:
Fig. 1 diagrammatically illustrates the applied field of data cache method and data buffer storage according to the embodiment of the present disclosure
Scape；
Fig. 2 diagrammatically illustrates the flow chart of the data cache method according to the embodiment of the present disclosure；
Fig. 3 diagrammatically illustrates data content in the first disk areas of acquisition according to one embodiment of the disclosure and changes
Data method flow diagram；
Fig. 4 diagrammatically illustrates data content in the first disk areas of acquisition according to another embodiment of the disclosure and becomes
The method flow diagram of the data of change；
Fig. 5 diagrammatically illustrates data content in the first disk areas of acquisition according to the another embodiment of the disclosure and becomes
The method flow diagram of the data of change；
Fig. 6 diagrammatically illustrates the method flow diagram cached again to partial data according to one embodiment of the disclosure；
Fig. 7 diagrammatically illustrates the block diagram of the data buffer storage device according to the embodiment of the present disclosure；And
Fig. 8 diagrammatically illustrates the block diagram of the computer system for being adapted for carrying out data buffer storage according to the embodiment of the present disclosure.
Specific embodiment
Hereinafter, will be described with reference to the accompanying drawings embodiment of the disclosure.However, it should be understood that these descriptions are only exemplary
, and it is not intended to limit the scope of the present disclosure.In the following detailed description, to elaborate many specific thin convenient for explaining
Section is to provide the comprehensive understanding to the embodiment of the present disclosure.It may be evident, however, that one or more embodiments are not having these specific thin
It can also be carried out in the case where section.In addition, in the following description, descriptions of well-known structures and technologies are omitted, to avoid
Unnecessarily obscure the concept of the disclosure.
Term as used herein is not intended to limit the disclosure just for the sake of description specific embodiment.It uses herein
The terms "include", "comprise" etc. show the presence of the feature, step, operation and/or component, but it is not excluded that in the presence of
Or add other one or more features, step, operation or component.
There are all terms (including technical and scientific term) as used herein those skilled in the art to be generally understood
Meaning, unless otherwise defined.It should be noted that term used herein should be interpreted that with consistent with the context of this specification
Meaning, without that should be explained with idealization or excessively mechanical mode.
It, in general should be according to this using statement as " at least one in A, B and C etc. " is similar to
Field technical staff is generally understood the meaning of the statement to make an explanation (for example, " system at least one in A, B and C "
Should include but is not limited to individually with A, individually with B, individually with C, with A and B, with A and C, have B and C, and/or
System etc. with A, B, C).Using statement as " at least one in A, B or C etc. " is similar to, generally come
Saying be generally understood the meaning of the statement according to those skilled in the art to make an explanation (for example, " having in A, B or C at least
One system " should include but is not limited to individually with A, individually with B, individually with C, with A and B, have A and C, have
B and C, and/or the system with A, B, C etc.).
Shown in the drawings of some block diagrams and/or flow chart.It should be understood that some sides in block diagram and/or flow chart
Frame or combinations thereof can be realized by computer program instructions.These computer program instructions can be supplied to general purpose computer,
The processor of special purpose computer or other programmable data processing units, so that these instructions are when executed by this processor can be with
Creation is for realizing function/operation device illustrated in these block diagrams and/or flow chart.The technology of the disclosure can be hard
The form of part and/or software (including firmware, microcode etc.) is realized.In addition, the technology of the disclosure, which can be taken, is stored with finger
The form of computer program product on the computer readable storage medium of order, the computer program product is for instruction execution system
System uses or instruction execution system is combined to use.
When carrying out big data operation by computing engines, in order to improve data-handling efficiency, it will usually by some data
(for example, data to be used are needed in repetitive operation or interative computation) caches in memory, to improve treatment effeciency.The disclosure
Embodiment provides a kind of data cache method and data buffer storage.The data cache method includes: to obtain the first caching number
Every data in is cached to the storage corresponding relationship of memory from the first disk areas, wherein this first it is data cached for
First moment was stored in the caching of the data in first disk areas；It obtains and is counted in first disk areas within first period
According to changed one or more data of content, which is when being continued until current after first moment
Between during；Based on the storage corresponding relationship, the cache information of one or more data is determined；And based on this one or more
The cache information of data caches one or more data again, first data cached to update this.
Data cache method and device according to an embodiment of the present disclosure, can be only by the first magnetic when carrying out buffer update
The changed partial data of data content is cached again in disk area.With caching all must be by the every time in the prior art
Total data in one disk areas caches again to be compared, can be big according to the data cache method of the embodiment of the present disclosure and device
Width reduces the data volume that caches every time, and the time needed for can shortening caching every time, in turn, can reduce and adjacent delay twice
The time interval deposited, real-time, substantially increases in practical business and imitates to the caching of the first disk areas with improving data buffer storage
Rate.
Fig. 1 diagrammatically illustrates the application scenarios of data cache method and data buffer storage according to the embodiment of the present disclosure
100.It should be noted that being only the example that can apply the application scenarios of the embodiment of the present disclosure shown in Fig. 1, to help this field
Technical staff understands the technology contents of the disclosure, but be not meant to the embodiment of the present disclosure may not be usable for other equipment, system,
Environment or scene.
As shown in Figure 1, the application scenarios 100 include the first disk areas 101, memory 102 and computing engines 103.The
Data buffer storage in one disk areas 101 is into memory 102, and the data content in the first disk areas 101 becomes
When change, changed partial data can be cached again, to realize the data and memory in the first disk areas 101
Data cached synchronization in 102.Computing engines 103 can read the data cached in memory 102 to carry out data processing.Meter
Calculating engine 103 for example can be spark.In some embodiments, computing engines 103 can also be from the first disk areas 101
Read data.
First disk areas 101 for example can be the storage region of various businesses database, wherein various businesses database
Such as can be relevant database (for example, MySQL database, oracle database or SQL server database etc.),
It is also possible to non-relational database.Correspondingly, the data in the first disk areas 101 can be in various businesses database
Data.Alternatively, in further embodiments, the first disk areas 101 for example can be the corresponding data storage area of data warehouse
Domain, such as distributed file system (such as hadoop distributed file system).Correspondingly, the first disk areas
Data in 101 for example can be hive table in distributed file system etc..
It, can be slow from the first disk areas 101 according to data according to the data cache method and device of the embodiment of the present disclosure
The storage corresponding relationship being stored in memory 102, it is right when the variation of the data content of the partial data in the first disk areas 101
The part that should be updated in memory 102 is data cached, is realized in the data in the first disk areas 101 and memory 102 with this
Data cached synchronization.Such as when caching, the data A in Fig. 1 in the first disk areas 101 is cached in data block a, number
It has been cached in data block b according to B, data C has been cached in data block c.In accordance with an embodiment of the present disclosure, it is assumed that when the first disk
Region 101 only has data A when changing, can be only data cached in more new data block a.In can thus fast implementing
Data volume when depositing the synchronous of the data in data cached and the first disk areas 101 in 102, and updating caching obviously subtracts
It is small, data buffer storage ground real-time is improved, and then substantially increase the timeliness that the computing engines such as spark 103 carry out business diagnosis
And accuracy rate.
Fig. 2 diagrammatically illustrates the flow chart of the data cache method according to the embodiment of the present disclosure.
As shown in Fig. 2, the data cache method may include operation S210~operation S240.
Operation S210, obtain first it is data cached in every data be cached to memory 102 from the first disk areas 101
Storage corresponding relationship, wherein this is first data cached for the data being stored in the first disk areas 101 at the first moment
Caching.
Specifically, for example, can recorde this first it is data cached in all data association messages, including every data is
Original storage information in one disk areas 101 and the cache information in memory 102 etc., to obtain, the storage is corresponding to be closed
System.Wherein, original storage information for example can be, and every data (such as data A, B, C shown in FIG. 1) is in the first disk areas
Storage location information in 101, store path information, every data including the database where every data or tables of data exist
Storage location (for example, row or column information), storing format informations in database or tables of data etc..For example, data A is in the first magnetic
It is stored in disk area 101 in a hive table.So the raw information of data A can be including the store path of the hive table
The row information and column field information etc. of information and data A in the hive table.Cache information for example can be, every number
According to which data block in which subregion or subregion being buffered in memory 102 and cache the information such as format.For example, figure
Data A is cached in data block a in 1, and data B is cached in data block b, and data C is cached in data block c.In turn, according to every
The original storage information and cache information of data, so that it may set up every data from the first disk areas 101 to memory 102
In storage corresponding relationship.
Operation S220, obtain within first period in the first disk areas 101 data content it is changed one or
A plurality of data, the first period are during being continued until the current time after first moment.The first period is phase
The time interval that neighbour caches twice, specific time span can be determined according to business needs.
Specifically, for example, the data situation in the first period in the first disk areas 101 can be monitored dynamically,
To obtain one or more data.Wherein, data content change including data content update, newly increase data or
Data are deleted.For example, it is desired to which the data of caching are the data in a sales volume tables of data in the first disk areas 101.Number
According to content update, some or all field (for example, sales volume field) in the sales volume tables of data in certain data can be
Value is updated, for example, the sales volume of certain product increases as time goes by, and then the numerical value of sales volume field can be updated.
Data are newly increased, such as when a product in new listing, one can be increase accordingly in the sales volume tables of data for recording this newly
The sales volume data of the product of listing.Data are deleted, such as when a product of undercarriage, are deleted and be used for from the sales volume tables of data
Record the sales volume data of the product of the undercarriage.
In operation S230, it is based on the storage corresponding relationship, determines caching letter of one or more data in memory 102
Breath.For example, determining one or more data in memory 102 according to the original storage information of one or more data
Cache information.
One or more data are cached again based on the cache information of one or more data in operation S240,
It is first data cached to update this.To changed partial data again buffer update, when specific implementation may include coarse grain
Degree updates and fine granularity updates two class situations.Coarseness updates, such as can be one or more data institute in memory
Data entirety buffer update in the data subregion or deblocking of category.Fine granularity updates, such as can be in memory 102
Orient the information such as specific storage location of every data in the one or moreth data, then in memory 102 to this
Every data in one or more data is cached again respectively.
In accordance with an embodiment of the present disclosure, on the one hand reduce data processing amount when each caching, on the other hand due to every
The data volume of secondary buffer update is obviously reduced, so can also significantly shorten compared with the existing technology it is adjacent cache twice when
Between be spaced, improve data cached real-time.In this way, it is possible to effectively alleviate since data buffer storage draws to calculating not in time
The influence of 103 carry out data processing bring hysteresis qualitys is held up, thus the accuracy and timeliness of business diagnosis can be greatly improved.
Fig. 3, which is diagrammatically illustrated, obtains number in the first disk areas 101 in the operation S220 according to one embodiment of the disclosure
According to the method flow diagram of the changed data of content.
As shown in figure 3, in accordance with an embodiment of the present disclosure, operation S220 may include operation S301 and operation S302.
In operation S301, the data query operation executed in the first period to the first disk areas 101 is monitored.
In operation S302, it is based on the data query operation, obtains one or more data.
It is illustrated with the data instance that the data for needing to cache in the first disk areas 101 are mySQL databases.Fig. 3
Method flow specific implementation, such as can be in the interface that computing engines 103 (for example, spark) are docked with mySQL insert
Enter monitoring module, monitors the operation to the data in mySQL.Then according in the library mySQL data manipulation instruction (for example,
Update, delete, insert operational order), it gets and is changed with the presence or absence of data variation and which data.
Fig. 4 is diagrammatically illustrated to be obtained in the first disk areas 101 in the operation S220 according to another embodiment of the disclosure
The method flow diagram of the changed data of data content.
As shown in figure 4, in accordance with an embodiment of the present disclosure, operation S220 may include operation S401~operation S403.
In operation S401, the record day for being recorded in the operation executed in the first period to the first disk areas 101 is obtained
Will.
In operation S402, the record log is analyzed, acquisition makes the changed operation information of data content.
In operation S403, it is based on the operation information, obtains one or more data.
Equally with the data instance of mySQL database.For example, can be monitored in mySQL database by spark
The variation of OPlog (Operation log, record log) file.OPlog file is the log that data manipulation is recorded in mySQL.
By reading or monitor the OPlog log, therefrom search whether there are specific operational order (such as update, delete,
The operational orders such as insert), so that it may it determines whether there is data variation and which data is changed.
Fig. 5 is diagrammatically illustrated to be obtained in the first disk areas 101 in the operation S220 according to the another embodiment of the disclosure
The method flow diagram of the changed data of data content
As shown in figure 5, in accordance with an embodiment of the present disclosure, operation S220 may include operation S501 and operation S502.
In operation S501, the newest content that record executes every data in first disk areas 101 in real time
The temporal information of change operation.
In operation S502, it is based on the temporal information, obtains the time of newest content alteration operation in the first period
Interior total data, to obtain one or more data.
Equally with the data instance of mySQL database.For example, can be to every in the tables of data for needing to cache in mySQL
Data beat timestamp.For example, it may be increasing the timestamp of column record data change in the tables of data of mySQL database.
Data change is once with regard to corresponding record once current time.For example, recording increased timestamp when newly-increased data, work as update
Also the timestamp that update is recorded when data, the timestamp of record deletion when deleting data.In this way, it is assumed that in 10:00:00 pair
Data synchronization caching in mySQL database is excessively primary, when (i.e. 10:00:05) is cached again after 5s, passes through acquisition time
Data of the stamp between 10:00:00-10:00:05, can be obtained one or more data.
Fig. 6 diagrammatically illustrates the method cached again in the operation S240 according to one embodiment of the disclosure to partial data
Flow chart.
As shown in fig. 6, in accordance with an embodiment of the present disclosure, operation S240 may include operation S601 and operation S602.Specifically
For, Fig. 6 illustrates the method flow that buffer update is carried out in such a way that coarseness updates.
Determine one or more data in memory based on the cache information of one or more data in operation S601
Data subregion or deblocking belonging in 102.
In operation S602, by the currently stored number corresponding with the data subregion or deblocking in the first disk areas 101
It is cached again according to whole.
Specifically, when if it is including newly-increased data in one or more data, such as it is true by Hash calculation etc.
The subregion where the newly-increased data is made, and the total data that then this can be increased newly in the subregion where data all caches
It updates.Alternatively, if when one or more data include the data or deleted data of content update, it is interior according to this
The cache information for holding the data or deleted data that update, by the total data in corresponding data subregion or deblocking
All carry out buffer update.
According to another embodiment of the present disclosure, operation S240 is also possible to fine granularity update as previously described, i.e., by this
Or each data in a plurality of data caches again respectively according to corresponding cache information.For example, when one or more number
When according to including the content update data of legacy data in the first disk areas 101, substituted with the content update data in memory 102
The legacy data of middle caching.Alternatively, for example, when one or more data include increasing newly in the first disk areas 101
When newly-increased data, increase the caching of the newly-increased data in memory 102.Alternatively, for example, when one or more data include from
When deleting the initial data of content in the first disk areas 101, the initial data cached in memory 102 is deleted.
Fig. 7 diagrammatically illustrates the block diagram of the data buffer storage device 700 according to the embodiment of the present disclosure.
As shown in fig. 7, in accordance with an embodiment of the present disclosure, data buffer storage device 700 include corresponding relationship obtain module 710,
Data in magnetic disk variation obtains module 720, caching change determining module 730 and buffer update module 740.According to the disclosure
Embodiment, the data buffer storage device 700 can be used to implement the data cache method according to the embodiment of the present disclosure.
Corresponding relationship obtain module 710 be used to obtain first it is data cached in every data from the first disk areas 101
It is cached to the storage corresponding relationship of memory 102, wherein this is first data cached to be stored in the first disk region at the first moment
The caching (operation S210) of data in domain 101.
Data in magnetic disk variation obtains module 720 and is used to obtain within first period data content in the first disk areas 101
Changed one or more data, the first period are during being continued until the current time after first moment
(operation S220).
Caching change determining module 730 is used to be based on the storage corresponding relationship, determines one or more data in memory
Cache information (operation S230) in 102.
Buffer update module 740 is used for the cache information based on one or more data, by one or more data
Again it caches, it is first data cached to update this.
In accordance with an embodiment of the present disclosure, data in magnetic disk variation obtains module 720 and is specifically used for: it is right in the first period to monitor
The data query operation (operation S301) that first disk areas 101 executes；And it is based on the data query operation, obtain this
Or a plurality of data (operation S302).
Alternatively, in accordance with an embodiment of the present disclosure, business change information obtains mould 720 and is specifically used for: acquisition be recorded in this
The record log (operation S401) for the operation that first disk areas 101 is executed during one；The record log is analyzed, acquisition makes
The changed operation information of data content (operation S402)；And it is based on the operation information, obtain one or more data
(operation S403).
Alternatively, in accordance with an embodiment of the present disclosure, data in magnetic disk variation obtains module 720 and is specifically used for: record is to the in real time
The temporal information (operation S501) for the newest content alteration operation that every data in one disk areas 101 executes；And
Based on the temporal information, total data of the time of newest content alteration operation within the first period is obtained, to obtain
One or more data (operation S502).
In accordance with an embodiment of the present disclosure, buffer update module 740 is specifically used for: the caching based on one or more data
Information determines one or more data data subregion affiliated in memory 102 or deblocking (operation S601), and will
Whole caching (the operation again of the currently stored data corresponding with the data subregion or deblocking in first disk areas 101
S602)。
Alternatively, in accordance with an embodiment of the present disclosure, buffer update module 740 is specifically used for will be in one or more data
Each data caches again respectively according to corresponding cache information.In accordance with an embodiment of the present disclosure, by one or more number
According to corresponding cache information, caching includes: when one or more data include first to each data in again respectively
In disk areas 101 when the content update data of legacy data, it is somebody's turn to do with what content update data substitution cached in memory 102
Legacy data；When one or more data include the newly-increased data increased newly in the first disk areas 101, in memory 102
The middle caching for increasing the newly-increased data；And when one or more data include deleting content from the first disk areas 101
Initial data when, delete the initial data that caches in memory 102.
According to the data buffer storage device 700 of the embodiment of the present disclosure, data processing when each caching on the one hand can be reduced
Amount, on the other hand can shorten the adjacent time interval cached twice, improve the real-time of data cached synchronization.Can specifically it join
The description of Fig. 2~Fig. 6 is examined, the disclosure repeats no more.
It is module according to an embodiment of the present disclosure, submodule, unit, any number of or in which any more in subelement
A at least partly function can be realized in a module.It is single according to the module of the embodiment of the present disclosure, submodule, unit, son
Any one or more in member can be split into multiple modules to realize.According to the module of the embodiment of the present disclosure, submodule,
Any one or more in unit, subelement can at least be implemented partly as hardware circuit, such as field programmable gate
Array (FPGA), programmable logic array (PLA), system on chip, the system on substrate, the system in encapsulation, dedicated integrated electricity
Road (ASIC), or can be by the hardware or firmware for any other rational method for integrate or encapsulate to circuit come real
Show, or with any one in three kinds of software, hardware and firmware implementations or with wherein any several appropriately combined next reality
It is existing.Alternatively, can be at least by part according to one or more of the module of the embodiment of the present disclosure, submodule, unit, subelement
Ground is embodied as computer program module, when the computer program module is run, can execute corresponding function.
For example, corresponding relationship obtains module 710, data in magnetic disk variation obtains module 720, caching changes determining module 730,
And any number of in buffer update module 740 may be incorporated in a module and realize or any one mould therein
Block can be split into multiple modules.Alternatively, at least partly function of one or more modules in these modules can be with it
He combines at least partly function of module, and realizes in a module.In accordance with an embodiment of the present disclosure, corresponding relationship obtains
Module 710, data in magnetic disk variation obtain in module 720, caching change determining module 730 and buffer update module 740 extremely
Few one can at least be implemented partly as hardware circuit, such as field programmable gate array (FPGA), programmable logic battle array
(PLA), system on chip, the system on substrate, the system in encapsulation, specific integrated circuit (ASIC) are arranged, or can be by electricity
Road carries out the hardware such as any other rational method that is integrated or encapsulating or firmware to realize, or with software, hardware and firmware
Any one in three kinds of implementations several appropriately combined is realized with wherein any.Alternatively, corresponding relationship obtains module
710, data in magnetic disk variation obtains at least one in module 720, caching change determining module 730 and buffer update module 740
It is a to be at least implemented partly as computer program module, when the computer program module is run, phase can be executed
The function of answering.
Fig. 8 diagrammatically illustrates the frame of the computer system 800 for being adapted for carrying out data buffer storage according to the embodiment of the present disclosure
Figure.Computer system 800 shown in Fig. 8 is only an example, should not function and use scope band to the embodiment of the present disclosure
Carry out any restrictions.
As shown in figure 8, computer system 800 includes processor 810 and computer readable storage medium 820.The calculating
Machine system 800 can execute the data cache method according to the embodiment of the present disclosure.
Specifically, processor 810 for example may include general purpose microprocessor, instruction set processor and/or related chip group
And/or special microprocessor (for example, specific integrated circuit (ASIC)), etc..Processor 810 can also include using for caching
The onboard storage device on way.Processor 810 can be the different movements for executing the method flow according to the embodiment of the present disclosure
Single treatment unit either multiple processing units.
Computer readable storage medium 820, such as can be non-volatile computer readable storage medium, specific example
Including but not limited to: magnetic memory apparatus, such as tape or hard disk (HDD)；Light storage device, such as CD (CD-ROM)；Memory, such as
Random access memory (RAM) or flash memory；Etc..
Computer readable storage medium 820 may include computer program 821, which may include generation
Code/computer executable instructions execute processor 810 according to the embodiment of the present disclosure
Data cache method or its any deformation.
Computer program 821 can be configured to have the computer program code for example including computer program module.Example
Such as, in the exemplary embodiment, the code in computer program 821 may include one or more program modules, for example including
821A, module 821B ....It should be noted that the division mode and number of module are not fixation, those skilled in the art can
To be combined according to the actual situation using suitable program module or program module, when these program modules are combined by processor 810
When execution, processor 810 is executed according to the data cache method of the embodiment of the present disclosure or its any deformation.
According to an embodiment of the invention, corresponding relationship obtains module 710, data in magnetic disk variation obtains module 720, caching becomes
More at least one of determining module 730 and buffer update module 740 can be implemented as the computer journey with reference to Fig. 8 description
Corresponding operating described above may be implemented when being executed by processor 810 in sequence module.
The disclosure additionally provides a kind of computer readable storage medium, which can be above-mentioned reality
It applies included in equipment/device/system described in example；Be also possible to individualism, and without be incorporated the equipment/device/
In system.Above-mentioned computer readable storage medium carries one or more program, when said one or multiple program quilts
When execution, the data cache method according to the embodiment of the present disclosure is realized.
In accordance with an embodiment of the present disclosure, computer readable storage medium can be non-volatile computer-readable storage medium
Matter, such as can include but is not limited to: portable computer diskette, hard disk, random access storage device (RAM), read-only memory
(ROM), erasable programmable read only memory (EPROM or flash memory), portable compact disc read-only memory (CD-ROM), light
Memory device, magnetic memory device or above-mentioned any appropriate combination.In the disclosure, computer readable storage medium can
With to be any include or the tangible medium of storage program, the program can be commanded execution system, device or device use or
Person is in connection.
Flow chart and block diagram in attached drawing are illustrated according to the system of the various embodiments of the disclosure, method and computer journey
The architecture, function and operation in the cards of sequence product.In this regard, each box in flowchart or block diagram can generation
A part of one module, program segment or code of table, a part of above-mentioned module, program segment or code include one or more
Executable instruction for implementing the specified logical function.It should also be noted that in some implementations as replacements, institute in box
The function of mark can also occur in a different order than that indicated in the drawings.For example, two boxes succeedingly indicated are practical
On can be basically executed in parallel, they can also be executed in the opposite order sometimes, and this depends on the function involved.Also it wants
It is noted that the combination of each box in block diagram or flow chart and the box in block diagram or flow chart, can use and execute rule
The dedicated hardware based systems of fixed functions or operations is realized, or can use the group of specialized hardware and computer instruction
It closes to realize.
It will be understood by those skilled in the art that the feature recorded in each embodiment and/or claim of the disclosure can
To carry out multiple combinations and/or combination, even if such combination or combination are not expressly recited in the disclosure.Particularly, exist
In the case where not departing from disclosure spirit or teaching, the feature recorded in each embodiment and/or claim of the disclosure can
To carry out multiple combinations and/or combination.All these combinations and/or combination each fall within the scope of the present disclosure.
Although the disclosure, art technology has shown and described referring to the certain exemplary embodiments of the disclosure
Personnel it should be understood that in the case where the spirit and scope of the present disclosure limited without departing substantially from the following claims and their equivalents,
A variety of changes in form and details can be carried out to the disclosure.Therefore, the scope of the present disclosure should not necessarily be limited by above-described embodiment,
But should be not only determined by appended claims, also it is defined by the equivalent of appended claims.
Claims (10)
1. a kind of data cache method, comprising:
Obtain first it is data cached in every data the storage corresponding relationship of memory is cached to from the first disk areas, wherein institute
State the first data cached caching for the data being stored in first disk areas at the first moment；
Obtain changed one or more data of data content in first disk areas within first period, described the
For during being continued until the current time after first moment during one；
Based on the storage corresponding relationship, the cache information of one or more data in the memory is determined；And
Based on the cache information of one or more data, one or more data are cached again, described in updating
First is data cached.
2. data cache method according to claim 1, wherein obtain within first period in first disk areas
Changed one or more data of data content, comprising:
Monitor the data query operation executed in the first period to first disk areas；And
Inquiry operation based on the data obtains one or more data.
3. data cache method according to claim 1, wherein obtain within first period in first disk areas
Changed one or more data of data content, comprising:
Obtain the record log for being recorded in the operation executed in the first period to first disk areas；
The record log is analyzed, acquisition makes the changed operation information of data content；And
Based on the operation information, one or more data are obtained.
4. data cache method according to claim 1, wherein acquisition first disk region within first period
Changed one or more data of data content include: in domain
Record believes the time for the newest content alteration operation that every data in first disk areas executes in real time
Breath；
Based on the temporal information, whole numbers of the time of newest content alteration operation within the first period are obtained
According to obtain one or more data.
5. data cache method according to claim 1, wherein the cache information based on one or more data,
One or more data are cached again, comprising:
Based on the cache information of one or more data, determine one or more data in the memory belonging to
Data subregion or deblocking；And
The currently stored data corresponding with the data subregion or deblocking in first disk areas are integrally delayed again
It deposits.
6. data cache method according to claim 1, wherein the cache information based on one or more data,
One or more data are cached again, comprising:
Each data in one or more data is cached again respectively according to corresponding cache information.
7. data cache method according to claim 6, wherein by each data in one or more data
It is cached again respectively according to corresponding cache information, comprising:
When one or more data include the content update data of legacy data in first disk areas, with described
Content update data substitute the legacy data cached in the memory；
When one or more data include the newly-increased data increased newly in first disk areas, in the memory
Increase the caching of the newly-increased data；And
When one or more data include deleting the initial data of content from first disk areas, described in deletion
The initial data cached in memory.
8. a kind of data buffer storage device, comprising:
Corresponding relationship obtain module, for obtain first it is data cached in every data be cached to memory from the first disk areas
Storage corresponding relationship, wherein described first is data cached for the number being stored in first disk areas at the first moment
According to caching；
Data in magnetic disk variation obtains module, and for obtaining, data content becomes in first disk areas within first period
One or more data changed, the first period are during being continued until the current time after first moment；
Caching change determining module determines one or more data described interior for being based on the storage corresponding relationship
Cache information in depositing；And
Buffer update module, for the cache information based on one or more data, by the one or more data weight
New caching, it is data cached to update described first.
9. data buffer storage device according to claim 8, wherein the buffer update module is specifically used for:
Based on the cache information of one or more data, determine one or more data in the memory belonging to
Data subregion or deblocking；And
The currently stored data corresponding with the data subregion or deblocking in first disk areas are integrally delayed again
It deposits.
10. data buffer storage device according to claim 8, wherein the buffer update module is specifically used for:
Each data in one or more data is cached again respectively according to corresponding cache information.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910226551.2A CN109981774B (en) | 2019-03-22 | 2019-03-22 | Data caching method and data caching device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910226551.2A CN109981774B (en) | 2019-03-22 | 2019-03-22 | Data caching method and data caching device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109981774A true CN109981774A (en) | 2019-07-05 |
| CN109981774B CN109981774B (en) | 2021-02-19 |
Family
ID=67080320
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910226551.2A Active CN109981774B (en) | 2019-03-22 | 2019-03-22 | Data caching method and data caching device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109981774B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111611287A (en) * | 2020-06-17 | 2020-09-01 | 北京商越网络科技有限公司 | Cache data updating method |
| CN113626458A (en) * | 2021-08-19 | 2021-11-09 | 咪咕数字传媒有限公司 | High-concurrency data updating method, device, equipment and computer storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103853727A (en) * | 2012-11-29 | 2014-06-11 | 深圳中兴力维技术有限公司 | Method and system for improving large data volume query performance |
| CN105701190A (en) * | 2016-01-07 | 2016-06-22 | 深圳市金证科技股份有限公司 | Data synchronizing method and device |
| US20180210795A1 (en) * | 2015-09-25 | 2018-07-26 | Huawei Technologies Co.,Ltd. | Data backup method and data processing system |
-
2019
- 2019-03-22 CN CN201910226551.2A patent/CN109981774B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103853727A (en) * | 2012-11-29 | 2014-06-11 | 深圳中兴力维技术有限公司 | Method and system for improving large data volume query performance |
| US20180210795A1 (en) * | 2015-09-25 | 2018-07-26 | Huawei Technologies Co.,Ltd. | Data backup method and data processing system |
| CN105701190A (en) * | 2016-01-07 | 2016-06-22 | 深圳市金证科技股份有限公司 | Data synchronizing method and device |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111611287A (en) * | 2020-06-17 | 2020-09-01 | 北京商越网络科技有限公司 | Cache data updating method |
| CN111611287B (en) * | 2020-06-17 | 2023-10-03 | 北京商越网络科技有限公司 | Cache data updating method |
| CN113626458A (en) * | 2021-08-19 | 2021-11-09 | 咪咕数字传媒有限公司 | High-concurrency data updating method, device, equipment and computer storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109981774B (en) | 2021-02-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11030189B2 (en) | Maintaining up-to-date materialized views for time-series database analytics | |
| US10831779B2 (en) | Seamless data migration across databases | |
| US20200372004A1 (en) | Indexing for evolving large-scale datasets in multi-master hybrid transactional and analytical processing systems | |
| US11132365B2 (en) | Query plan based on a data storage relationship | |
| CN107329982A (en) | A kind of big data parallel calculating method stored based on distributed column and system | |
| CN112534396A (en) | Diary watch in database system | |
| US9875272B1 (en) | Method and system for designing a database system for high event rate, while maintaining predictable query performance | |
| JP5233233B2 (en) | Information search system, information search index registration device, information search method and program | |
| US20160371328A1 (en) | Partition access method for query optimization | |
| JPWO2012131927A1 (en) | Computer system and data management method | |
| US10614069B2 (en) | Workflow driven database partitioning | |
| CN106716409A (en) | Method and system for adaptively building and updating column store database from row store database based on query demands | |
| US20180004783A1 (en) | Database object management for a shared pool of configurable computing resources | |
| Khayyat et al. | Fast and scalable inequality joins | |
| US20170270183A1 (en) | Sensor data management apparatus, sensor data management method, and computer program product | |
| Fang et al. | Dragoon: a hybrid and efficient big trajectory management system for offline and online analytics | |
| CN110309233A (en) | Method, apparatus, server and the storage medium of data storage | |
| CN104283866A (en) | Server system for providing current data and past data to clients | |
| CN109981774A (en) | Data cache method and data buffer storage | |
| US10884998B2 (en) | Method for migrating data records from a source database to a target database | |
| US20230161795A1 (en) | Time series data management systems and methods | |
| Zheng et al. | Timo: In‐memory temporal query processing for big temporal data | |
| US20210064617A1 (en) | Reducing temp size over a base table | |
| Tian et al. | Tinba: Incremental partitioning for efficient trajectory analytics | |
| US11841865B2 (en) | Database management system and associated methods |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |