CN103488772A - Method, system and equipment for caching files through external storage - Google Patents
Method, system and equipment for caching files through external storage Download PDFInfo
- Publication number
- CN103488772A CN103488772A CN201310452157.3A CN201310452157A CN103488772A CN 103488772 A CN103488772 A CN 103488772A CN 201310452157 A CN201310452157 A CN 201310452157A CN 103488772 A CN103488772 A CN 103488772A
- Authority
- CN
- China
- Prior art keywords
- file
- buffer memory
- document
- read
- index
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
- G06F16/134—Distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method, a system and equipment for caching files through an external storage. The method includes that data needed to be cached are stored in the external storage, and capacity of cached data is greatly increased; at the same time, a caching index is added to the cached data, so that during retrieving, the cached data can be quickly retrieved, and retrieving speed is quickened; during changing or first-time reading of files, for the files needed to be written in the cached data, reading is performed according to a conventional process; during reading, the files are stored in a caching sequence and written into the cached data through the caching sequence, conventional reading and caching writing-in are enabled to performed simultaneously; reading from the cached dat can be performed quickly during next-time reading, so that writing-reading speed is increased.
Description
Technical field
The present invention relates to the buffer memory of file, particularly a kind of method, system and equipment of buffer memory of file being carried out by external memory.
Background technology
Buffer memory is a kind of temporary file exchange area, and existing technical method is by buffer memory (RAM) or high-speed cache (cache), the information of recent visit to be stored temporarily, therefrom reads when needs are accessed, thereby realizes fast access, saves time.But the ubiquitous problem of RAM and high-speed cache is once that, outage, data cached will the disappearance, need to store again; Simultaneously, with respect to External memory equipments such as hard disks, RAM and high-speed cache cost are more expensive, are confined to the size of RAM and cache memory space, and data cached also have a restriction, development along with technology, the lifting of computing equipment calculated performance, the software program of operation is increasing, and needed spatial cache in service is also along with increase, therefore, need the restriction that the existing RAM of breakthrough and cache spatial cache are less.
In addition, RAM and cache, when data cached, do not consider the positional information of data cached storage, therefore, are reading requiredly when data cached, just need to successively find and be read, and have affected reading efficiency.
Summary of the invention
For addressing the above problem, the object of the present invention is to provide a kind of method, system and equipment of buffer memory of file being carried out by external memory, realize buffer memory by external memory, enlarge spatial cache, be data cached increase index list simultaneously, realize the purpose read fast.
The present invention solves the technical scheme that its problem adopts:
A kind of method, system and equipment of buffer memory of file being carried out by external memory, when file reading, whether detect this document is present in the buffer memory index, the buffer memory index is index list, this index list includes the index entry that is stored in data cached middle All Files, and the index entry of each file comprises the Last modification time of the file identification of this document, this document, the off-set value of this document in data cached and the size of this document.Detecting the file read whether in the buffer memory index time, need the retrieving files sign, take file identification as cryptographic hash be example, need to calculate the cryptographic hash of this file, if find identical cryptographic hash in the buffer memory index, illustrate this document is arranged in data cached, but need the file that further judgement is read whether whether consistent with this document of data cached middle storage, because, the file read may pass through modification, therefore, the Last modification time that needs the file that judgement reads whether with in the buffer memory index, record identical, if identical, illustrate that this file does not change, can directly from data cached, read this file, otherwise, illustrate that change has occurred read file, need to read according to conventional flow process this file, this file is write in buffer queue simultaneously, if do not find identical cryptographic hash in the buffer memory index, illustrate that this file is for reading for the first time, read this file according to old process and get final product,, this file is write in buffer queue simultaneously.
Write to file in buffer queue need to write to data cached in, to facilitate use next time, reduce and read the time, buffer queue is the queue of first in first out, read successively file wherein, after reading at every turn, judge whether this document is present in the buffer memory index, if exist, this file is described and reads for the first time, its be stored in data cached in, the size that needs the file that reads of judgement and data cached middle stored this document, if the former is less than or equal to the latter, the raw address that directly it is write to data cached middle stored this document gets final product, otherwise, need to open up at data cached end the file that read for storage in new space, on the contrary, if the file read is not present in the buffer memory index, is illustrated as and reads for the first time this file, directly at data cached end, open up new space, this file is write and gets final product.
The buffer memory index that the present invention is used for, buffer queue and data cachedly all by external memory, realize, the equipment that can be used in comprises computing machine, palm PC, smart machine etc., the present invention all can realize on these equipment.
The invention has the beneficial effects as follows:
The present invention adopts a kind of method, system and equipment of buffer memory of file being carried out by external memory, and the data that need buffer memory are stored among external memory, greatly increases data cached capacity, it is data cached increase buffer memory index simultaneously, when retrieval, retrieve rapidly, accelerate retrieval time; Change or read for the first time for file, need to write to data cached file, while reading, according to old process, read, and, when reading, these files are stored in buffer queue, and by these files, write to data cached by this buffer queue, routine is read with buffer memory to be write simultaneously and carries out, and, when reading next time, read more rapidly, thereby accelerate read or write speed from data cached.
The accompanying drawing explanation
Below in conjunction with accompanying drawing and example, the invention will be further described.
Fig. 1 is the process flow diagram of the method for the invention;
Fig. 2 is the schematic diagram of buffer memory index of the present invention;
Fig. 3 is buffer memory index of the present invention and data cached corresponding relation figure.
Embodiment
With reference to the process flow diagram of the method for the invention shown in Fig. 1, the present invention realizes the buffer memory of data by external memory, thereby realizes the fast reading and writing operation to file.Described method is specific as follows:
When file reading, at first in the buffer memory index, searched, the buffer memory index is data cached index list, this index list has recorded the index entry of data cached middle All Files, wherein, in data cached, the index entry of a file comprises the file identification of this document, the Last modification time of this document, the off-set value of this document in data cached and the size of this document, wherein, the file identification of file comprises filename, file ID, the combination of one or more in the file cryptographic hash etc., but for different file identifications, must keep consistent with in the buffer memory index.When file reading, by file identification, in the buffer memory index, searched, if find identical file identification, illustrate read file be stored in data cached in.But whether the file that needs judgement to read changes, if the file read changes, illustrate that the file be stored in data cached need to write again, otherwise the file read will be inconsistent.Therefore, the Last modification time that further needs the file that judgement reads whether with in the buffer memory index, record identical, if identical, illustrate that read file do not change, can directly from data cached, read corresponding file, on the contrary, the file that explanation is read is changed, and can not from data cached, read, and can only be read in a conventional manner, when reading, this document can be written in buffer queue.It should be noted that, routine reads with writing to buffer queue and walks abreast, and is independent of each other, to save time.
If, while searching in the buffer memory index by file identification, do not find identical file identification, illustrate that this document is for reading for the first time, directly carry out routine and read, this file is write in buffer queue simultaneously.
So far, the file of occur changing and reading for the first time for file, all be written in buffer queue, and buffer queue is the queue of first in first out, and the file that writes to queue, for the convenience that reads next time and quick, need to write to it data cached.Write fashionable, read successively the file in buffer queue, because file is divided into two kinds, a kind of is the file of changing, a kind of is the file read for the first time, for the file of changing, its file identification has been stored in the middle of the buffer memory index, the simultaneously data cached file of also having stored before changing, in order to save spatial cache, need this file of changing of judgement and data cached in the size of the front file of the change of having stored, if the former is larger, need to be its opening space at data cached end, and it is write data cached, on the contrary, can directly it be write to the original position of data cached middle stored change preceding document, with conserve space.For the file read for the first time, directly at data cached end opening space, the size in space equates with the file read for the first time, then it is write to data cached getting final product.
Finally, the file in buffer queue write to data cached after, need to be upgraded the buffer memory index, the index entry of the file that is about to write writes in the buffer memory index.
Schematic diagram with reference to the buffer memory index shown in Fig. 2, the buffer memory index comprises the index entry of different files, the index entry of each file comprises four, be respectively the file identification of this document, last modification time, off-set value and file size, file identification comprises cryptographic hash, filename etc., when using cryptographic hash, guarantee the consistance of computing method, last modification time can judge whether this document carried out modification, off-set value has indicated the position of this document in data cached, and its size has been determined the data area while reading.
With reference to the buffer memory index shown in Fig. 3 and data cached corresponding relation figure, the occupied space of buffer memory index is with respect to data cached very little, can convenience file search, save the time of retrieval, the corresponding file in data cached of each index entry that the buffer memory index comprises, when reading this document, can read fast.
The present invention realizes the buffer memory of file by external memory, can enlarge spatial cache, for larger file, also can realize buffer memory, as the data of client 10G; For data cached, set up a buffer memory index entry and guided, be equivalent to specify index position for each cache file, realize location fast, accelerate retrieval time; For the file of changing or the file read for the first time, they are write in the middle of a buffer queue, to carry out routine and read when writing, two processes are undertaken by different threads, accelerate reading speed; For the file in buffer queue, the operation by background program by its write to data cached in, to facilitate next time, read, save and read the time.
The invention provides a kind of system of buffer memory of file being carried out by external memory, comprising:
Detection module, detect read file during for file reading and whether be present in the buffer memory index, described buffer memory index is the index list for the index entry of record buffer memory data All Files, wherein, in data cached, the index entry of each file comprises the file identification of this document, Last modification time, the off-set value of this document in data cached and the size of this document of this document, and file identification comprises filename, cryptographic hash etc.;
Read module, if be present in the buffer memory index for the file read, the Last modification time of the file that reads of judgement whether with in the buffer memory index, record identical, if identical, according to the index entry of this document recorded in the buffer memory index, read the cache file identical with the institute file reading be stored in data cached, if it is not identical, read file is write in buffer queue, read flow process by routine simultaneously and read this document, if the file read is not present in the buffer memory index, directly read file is write in buffer queue, read flow process by routine simultaneously and read this document,
Cache module, for reading successively the file of buffer queue, and write to data cached by the file in buffer queue; And
Update module, for upgrading the buffer memory index, write to the buffer memory index by the index entry that writes to data cached file.
Wherein, when detecting read file during described file reading and whether being present in the buffer memory index, by the file identification by institute's file reading, in the buffer memory index, searched, if find identical file identification, the file that read is present in the buffer memory index, otherwise, be not present in the buffer memory index.
Wherein, described buffer queue is fifo queue.
Wherein, described file in buffer queue is written to data cached being specially:
Judge whether this document is present in the buffer memory index, if while existing, relatively this document and the size that is stored in this document in data cached, if the former is less than or equal to the latter, read the content of this document, its content is write to data cached specified position, if the former is greater than the latter, at data cached end, distribute the space identical with this document size, and this document is write in the space of distribution; On the contrary, if when this document is not present in the buffer memory index, directly at data cached end, distributes the space identical with this document size, and this document is write in the space of distribution.
Described system has adopted the content in the above method, specifically referring to described in method.
The invention provides a kind of equipment of buffer memory that file carried out by external memory, described equipment has adopted above-described system, and described equipment comprises computer equipment, can realize the present invention by hard disk, USB flash disk etc., also comprise Intelligent mobile equipment, as smart mobile phone, palm PC etc.
The above, be preferred embodiment of the present invention, and the present invention is not limited to above-mentioned embodiment, as long as it reaches technique effect of the present invention with identical means, all should belong to protection scope of the present invention.
Claims (13)
1. one kind is carried out the method for buffer memory to file by external memory, it is characterized in that, comprising:
Whether detect read file during file reading and be present in the buffer memory index, described buffer memory index is the index list for the index entry of record buffer memory data All Files;
If the file read is present in the buffer memory index, the Last modification time of the file that reads of judgement whether with in the buffer memory index, record identical, if identical, according to the index entry of this document recorded in the buffer memory index, read the identical cache file of the file with read be stored in data cached, if it is not identical, read file is write in buffer queue, read flow process by routine simultaneously and read this document, if the file read is not present in the buffer memory index, directly read file is write in buffer queue, read flow process by routine simultaneously and read this document, and
Read successively the file in buffer queue, and write to data cached by the file in buffer queue.
2. method according to claim 1, is characterized in that, described method also comprises:
Upgrade the buffer memory index, the index entry that writes to data cached file is write to the buffer memory index.
3. method according to claim 1 and 2, it is characterized in that, described buffer memory index is as the index list of the index entry for record buffer memory data All Files, and the index entry of its each file comprises the Last modification time of the file identification of this document, this document, the off-set value of this document in data cached and the size of this document.
4. method according to claim 3, it is characterized in that, when detecting read file during described file reading and whether being present in the buffer memory index, by the file identification by institute's file reading, in the buffer memory index, searched, if find identical file identification, the file that read is present in the buffer memory index, otherwise, be not present in the buffer memory index.
5. method according to claim 3, is characterized in that, described buffer queue is fifo queue.
6. method according to claim 3, is characterized in that, described file in buffer queue is written to data cached being specially:
Judge whether this document is present in the buffer memory index, if while existing, relatively this document and the size that is stored in this document in data cached, if the former is less than or equal to the latter, read the content of this document, its content is write to data cached specified position, if the former is greater than the latter, at data cached end, distribute the space identical with this document size, and this document is write in the space of distribution; On the contrary, if when this document is not present in the buffer memory index, directly at data cached end, distributes the space identical with this document size, and this document is write in the space of distribution.
7. one kind is carried out the system of buffer memory to file by external memory, it is characterized in that, comprising:
Detection module, detect read file and whether be present in the buffer memory index during for file reading, described buffer memory index is the index list for the index entry of record buffer memory data All Files;
Read module, if be present in the buffer memory index for the file read, the Last modification time of the file that reads of judgement whether with in the buffer memory index, record identical, if identical, according to the index entry of this document recorded in the buffer memory index, read the cache file identical with the institute file reading be stored in data cached, if it is not identical, read file is write in buffer queue, read flow process by routine simultaneously and read this document, if the file read is not present in the buffer memory index, directly read file is write in buffer queue, read flow process by routine simultaneously and read this document, and
Cache module, for reading successively the file of buffer queue, and write to data cached by the file in buffer queue.
8. system according to claim 7, is characterized in that, described system also comprises:
Update module, for upgrading the buffer memory index, write to the buffer memory index by the index entry that writes to data cached file.
9. according to the described system of claim 7 or 8, it is characterized in that, described buffer memory index is as the index list of the index entry for record buffer memory data All Files, and the index entry of its each file comprises the Last modification time of the file identification of this document, this document, the off-set value of this document in data cached and the size of this document.
10. system according to claim 9, it is characterized in that, when detecting read file during described file reading and whether being present in the buffer memory index, by the file identification by institute's file reading, in the buffer memory index, searched, if find identical file identification, the file that read is present in the buffer memory index, otherwise, be not present in the buffer memory index.
11. system according to claim 9, is characterized in that, described buffer queue is fifo queue.
12. system according to claim 9, is characterized in that, described file in buffer queue is written to data cached being specially:
Judge whether this document is present in the buffer memory index, if while existing, relatively this document and the size that is stored in this document in data cached, if the former is less than or equal to the latter, read the content of this document, its content is write to data cached specified position, if the former is greater than the latter, at data cached end, distribute the space identical with this document size, and this document is write in the space of distribution; On the contrary, if when this document is not present in the buffer memory index, directly at data cached end, distributes the space identical with this document size, and this document is write in the space of distribution.
13. one kind is carried out the equipment of buffer memory to file by external memory, it is characterized in that, described equipment comprises the arbitrary described system of claim 7 to 12.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310452157.3A CN103488772A (en) | 2013-09-27 | 2013-09-27 | Method, system and equipment for caching files through external storage |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310452157.3A CN103488772A (en) | 2013-09-27 | 2013-09-27 | Method, system and equipment for caching files through external storage |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN103488772A true CN103488772A (en) | 2014-01-01 |
Family
ID=49828998
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310452157.3A Pending CN103488772A (en) | 2013-09-27 | 2013-09-27 | Method, system and equipment for caching files through external storage |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103488772A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104636464A (en) * | 2015-02-09 | 2015-05-20 | 腾讯科技（深圳）有限公司 | Method and device for having access to file |
| CN106815329A (en) * | 2016-12-29 | 2017-06-09 | 网易无尾熊（杭州）科技有限公司 | A kind of data cached update method and device |
| CN107038180A (en) * | 2016-08-23 | 2017-08-11 | 平安科技（深圳）有限公司 | Insurance application information updating method and management server |
| CN109634877A (en) * | 2018-12-07 | 2019-04-16 | 广州市百果园信息技术有限公司 | Flow implementation method, device, equipment and the storage medium of operation |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101169761A (en) * | 2007-12-03 | 2008-04-30 | 腾讯数码（天津）有限公司 | Large capacity cache implement method and storage system |
| CN101291347A (en) * | 2008-06-06 | 2008-10-22 | 中国科学院计算技术研究所 | Network storage system |
| CN102662459A (en) * | 2012-04-22 | 2012-09-12 | 复旦大学 | Method for reducing energy consumption of server by using mixed storage of solid-state drive and mechanical hard disk |
| CN102843396A (en) * | 2011-06-22 | 2012-12-26 | 中兴通讯股份有限公司 | Data writing and reading method and device in distributed caching system |
-
2013
- 2013-09-27 CN CN201310452157.3A patent/CN103488772A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101169761A (en) * | 2007-12-03 | 2008-04-30 | 腾讯数码（天津）有限公司 | Large capacity cache implement method and storage system |
| CN101291347A (en) * | 2008-06-06 | 2008-10-22 | 中国科学院计算技术研究所 | Network storage system |
| CN102843396A (en) * | 2011-06-22 | 2012-12-26 | 中兴通讯股份有限公司 | Data writing and reading method and device in distributed caching system |
| CN102662459A (en) * | 2012-04-22 | 2012-09-12 | 复旦大学 | Method for reducing energy consumption of server by using mixed storage of solid-state drive and mechanical hard disk |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104636464A (en) * | 2015-02-09 | 2015-05-20 | 腾讯科技（深圳）有限公司 | Method and device for having access to file |
| CN104636464B (en) * | 2015-02-09 | 2019-02-01 | 腾讯科技（深圳）有限公司 | The method and apparatus for accessing file |
| CN107038180A (en) * | 2016-08-23 | 2017-08-11 | 平安科技（深圳）有限公司 | Insurance application information updating method and management server |
| WO2018036152A1 (en) * | 2016-08-23 | 2018-03-01 | 平安科技（深圳）有限公司 | Method for updating insurance policy information, management server, and computer readable storage medium |
| CN106815329A (en) * | 2016-12-29 | 2017-06-09 | 网易无尾熊（杭州）科技有限公司 | A kind of data cached update method and device |
| CN109634877A (en) * | 2018-12-07 | 2019-04-16 | 广州市百果园信息技术有限公司 | Flow implementation method, device, equipment and the storage medium of operation |
| CN109634877B (en) * | 2018-12-07 | 2023-07-21 | 广州市百果园信息技术有限公司 | Method, device, equipment and storage medium for realizing stream operation |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9996466B2 (en) | Apparatus, system and method for caching compressed data | |
| US8595451B2 (en) | Managing a storage cache utilizing externally assigned cache priority tags | |
| CN107168657B (en) | Virtual disk hierarchical cache design method based on distributed block storage | |
| CN101673188B (en) | Data access method for solid state disk | |
| CN102782683B (en) | Buffer pool extension for database server | |
| CN101981551B (en) | Apparatus and method for cache utilization | |
| CN105242871A (en) | Data writing method and apparatus | |
| KR20090026296A (en) | Predictive data-loader | |
| CN103049224B (en) | By the methods, devices and systems of data importing physical tape | |
| CN103150395B (en) | Directory path analysis method of solid state drive (SSD)-based file system | |
| CN103985393B (en) | A kind of multiple optical disk data parallel management method and device | |
| US10489296B2 (en) | Quality of cache management in a computer | |
| CN101645043A (en) | Methods for reading and writing data and memory device | |
| CN103345449B (en) | A kind of fingerprint forecasting method towards data de-duplication technology and system | |
| CN100458736C (en) | NAND flash information extraction method and NAND flash automatic identification method | |
| CN103399823A (en) | Method, equipment and system for storing service data | |
| CN105787012A (en) | Method for improving small file processing capability of storage system and storage system | |
| CN103136215A (en) | Data read-write method and device of storage system | |
| CN103488772A (en) | Method, system and equipment for caching files through external storage | |
| SG126863A1 (en) | Recording apparatus | |
| CN100580669C (en) | Method for realizing cache memory relates to file allocation table on Flash storage medium | |
| US9524236B1 (en) | Systems and methods for performing memory management based on data access properties | |
| CN102981975A (en) | Management method of flash data and system thereof | |
| CN105654972A (en) | Exchangeable caddy and optical disc juke box-orientated fast self-checking method and device | |
| CN108958657B (en) | Data storage method, storage device and storage system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20140101 |