CN100530186C - Method and system for processing buffer - Google Patents

Method and system for processing buffer Download PDF

Info

Publication number
CN100530186C
CN100530186C CNB2006101671808A CN200610167180A CN100530186C CN 100530186 C CN100530186 C CN 100530186C CN B2006101671808 A CNB2006101671808 A CN B2006101671808A CN 200610167180 A CN200610167180 A CN 200610167180A CN 100530186 C CN100530186 C CN 100530186C
Authority
CN
China
Prior art keywords
cache
file
write
module
memory
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CNB2006101671808A
Other languages
Chinese (zh)
Other versions
CN101075241A (en
Inventor
王海波
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tencent Technology Shenzhen Co Ltd
Tencent Cloud Computing Beijing Co Ltd
Original Assignee
Tencent Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tencent Technology Shenzhen Co Ltd filed Critical Tencent Technology Shenzhen Co Ltd
Priority to CNB2006101671808A priority Critical patent/CN100530186C/en
Publication of CN101075241A publication Critical patent/CN101075241A/en
Application granted granted Critical
Publication of CN100530186C publication Critical patent/CN100530186C/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Landscapes

  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

A method for processing buffer storage includes seeking file in Memory Cache as per index when file is required to be obtained, fetching file in cache item if corresponding cache item is sought; or otherwise seeking file in Disk Cache as per index, setting up new item in Memory Cache if corresponding cache item is sought, writing file in cache item into set-up item and fetching said file; or otherwise downloading and obtaining said file from network, setting up mew item in Memory Cache and writing said file into set-up item cache item. The process system used for realizing said method is also disclosed.

Description

Method for caching and processing and caching process system
Technical field
The present invention relates to buffer memory (cache) technical field in the memory technology, particularly a kind of method for caching and processing and a kind of caching process system.
Background technology
Development along with Internet technology, a large amount of network users uses the WEB server on the access internet through browsers, the user is usually by the data file of HTTP(Hypertext Transport Protocol) mode to a URL correspondence of WEB server requests, the WEB server is made user's request by http protocol and being replied, and returns the data file of being asked.
Along with the increase of number of users, limited WEB server bandwidth is competitively used in the user capture of magnanimity, causes access speed to descend; Owing to the difference of operator on the internet or the difference of geographical environment, also can cause the speed of the visit WEB of certain customers server very slow simultaneously.
It is that a kind of mode by point-to-point (P2P) is quickened the method for user capture WEB server page that WEB quickens.The WEB data in server file that the user terminal buffer memory was visited recently.Other users obtain the data file that self needs by the mode of P2P there from the user terminal of visiting identical WEB server data file, and can use the HTTP mode to obtain data from the WEB server synchronously, quicken the visit to the WEB server.
Because the data file of WEB visit is most of smaller, and quantity is many, carry out metadata cache for the heap file of this form, its concurrent disk access can cause system overhead bigger, thereby influences acceleration effect.Simultaneously, because WEB quickens to download the cache file by the mode of P2P, safety issue also needs to consider.Therefore, how to manage the technical matters that buffer memory has become to be badly in need of solution effectively.
Summary of the invention
In view of this, the present invention proposes a kind of method for caching and processing, the buffer memory of user terminal in quickening in order to effective management WEB.Another object of the present invention is a kind of caching process of proposition system.
The present invention proposes a kind of method for caching and processing according to above-mentioned purpose, this method comprises:
A. when needs when network obtains file, search in the memory cache module according to index, if find the corresponding cache clauses and subclauses, then read file wherein, otherwise execution in step B; Described index is that the URL(uniform resource locator) of Web file (URL) is carried out the unified HASH value of length that obtains behind the Hash (HASH);
B. search in the disk buffering module according to index, if find the corresponding cache clauses and subclauses, then in the memory cache module, construct new clauses and subclauses, with the file in the corresponding cache clauses and subclauses in the described disk buffering module write construct in the cache entries, and read described file, otherwise execution in step C;
C. from network, download and obtain described file, the new cache entries of structure in the memory cache module, and in this cache entries, write described file;
This method further comprises: the cache entries in the memory cache module is write the disk buffering module;
When writing size less than the small documents of setting threshold, with its write to comprise the big or small classification of small documents identical with the small documents that will write and have the big file in enough spaces or it is write the big file of establishment;
When the described small documents of deletion, this small documents of deletion from the big file at its place, and when not having small documents in the described big file, delete described big file;
When reading described small documents, from the big file at its place, read this small documents.
In the memory cache module, further comprise the step of judging that the memory cache module is whether full among step B and the step C before the new cache entries of structure, if, then delete the cache entries that does not read at most in the memory cache module, carry out then and construct; Otherwise, directly carry out structure.
In the cache entries it is being write the disk buffering module described in the deletion memory cache module.
Describedly cache entries in the memory cache module is write the disk buffering module be: will not have accessed cache entries to write the disk buffering module in the memory cache module at most.
Before writing the disk buffering module, further comprise the step of judging that the disk buffering module is whether full, if, then delete the cache entries that is not modified at most in the disk buffering module, carry out writing then; Write otherwise directly carry out.
The described step that writes file in cache entries comprises: obtain the current accessed state of cache entries, when the current accessed state of cache entries is a WriteMode or when reading mode, forbid writing; When the current accessed state of cache entries is when not having user capture, in this cache entries, write file, and the current state of record buffer memory clauses and subclauses is a WriteMode; After writing end, the current state of record buffer memory is not for there being user capture.
The step that reads the file in the cache entries comprises: obtain the current accessed state of cache entries, when the current accessed state of cache entries is WriteMode, forbid reading; When the current accessed state of cache entries is when reading mode or not having user capture, read the file in the cache entries, and the number of users of reading of cache entries is added 1; After reading end, when the reading number of users and be 0 of cache entries, the current state of record buffer memory clauses and subclauses is not for there being user capture.
This method further comprises: when obtaining cache entries from the disk buffering module, read the wherein HASH value of file, and calculate the HASH value of the file that is read, if two HASH values are identical, it is legal then authenticating described file, is illegal otherwise authenticate described file; And/or authentication is legal by HTTP mode downloaded files; And/or, some or all of when the P2P mode file in download, download corresponding HASH value, and the HASH value of calculating institute downloaded files, if two HASH values are identical, it is legal then authenticating described file, is illegal otherwise authenticate described file.
The invention allows for a kind of caching process system, this system comprises memory cache module, disk buffering module and caching management module, and wherein: the memory cache module is used in internal memory memory buffers clauses and subclauses;
The disk buffering module is used in disk memory buffers clauses and subclauses;
Caching management module, be used for when needs when network obtains file, search in the memory cache module according to index, if find the corresponding cache clauses and subclauses then read wherein file, otherwise in the disk buffering module, search according to index, if find the corresponding cache clauses and subclauses, then in the memory cache module, construct new cache entries, file in the corresponding cache clauses and subclauses in the described disk buffering module is write in the cache entries of being constructed, and read described file, otherwise from network, download and obtain described file, the cache entries new in the memory cache module according to structure, and in this cache entries, write described file; Cache entries in the memory cache module is write the disk buffering module; When writing size less than the small documents of setting threshold, with its write to comprise the big or small classification of small documents identical with the small documents that will write and have the big file in enough spaces or it is write the big file of establishment; When the described small documents of deletion, this small documents of deletion from the big file at its place, and when not having small documents in the described big file, delete described big file; When reading described small documents, from the big file at its place, read this small documents; Described index is that the URL to the Web file carries out the unified HASH value of length that obtains behind the HASH.
From such scheme as can be seen, because the present invention's memory buffers clauses and subclauses in internal memory and disk, when needs obtain web page files, in the memory cache module, search earlier according to index, if find the corresponding cache clauses and subclauses, then read file wherein, if do not find then further in the disk buffering module, search, if find the corresponding cache clauses and subclauses, then in the memory cache module, construct new cache entries, file in the corresponding cache clauses and subclauses in the described disk buffering module is write in the cache entries of being constructed, thereby get access to described file, obtain described file otherwise from network, download, the cache entries new in the memory cache module then according to structure, and in this cache entries, write described file, pass through such scheme, the present invention has improved the speed of user capture buffer memory, reduced system overhead, has improved the speed that the user browses the WEB webpage.
Description of drawings
Fig. 1 is the structural representation of caching process system in the embodiment of the invention;
Fig. 2 is the structural representation of the cache clauses and subclauses in the embodiment of the invention;
The schematic flow sheet of Fig. 3 in the embodiment of the invention buffer memory being handled;
Fig. 4 is for writing the schematic flow sheet of cache clauses and subclauses in DiskCache in the embodiment of the invention;
Fig. 5 is the schematic flow sheet of visit cache clauses and subclauses in the embodiment of the invention;
Fig. 6 is for carrying out the schematic flow sheet of legitimacy authentication to the cache file in the embodiment of the invention;
Fig. 7 is the structural representation of big file in the embodiment of the invention;
Fig. 8 is for writing the schematic flow sheet of small documents in the embodiment of the invention;
Fig. 9 is the schematic flow sheet of deletion small documents in the embodiment of the invention.
Embodiment
For making the purpose, technical solutions and advantages of the present invention clearer, the present invention is described in more detail by the following examples.
Shown in Figure 1 is the structural representation of caching process system in the embodiment of the invention.With reference to Fig. 1, this system comprises caching management module (CacheMgr), memory cache module (MemoryCache) and disk buffering module (DiskCache).
Wherein, CacheMgr is used for managing MemoryCache and the data cached operations such as writing, read of DiskCache, and other module accesses buffer memory need be interface with CacheMgr all.The MemoryCache that is arranged in internal memory is respectively applied for the cache clauses and subclauses of stored memory and the cache clauses and subclauses (Entry) of disk with the DiskCache that is arranged in disk.As everyone knows, very fast according to the characteristics of access memory and visit disk for the visit of MemoryCache, and slower for the visit of DiskCache.
MemoryCache is made up of the cache clauses and subclauses that are stored in the internal memory, and the cache clauses and subclauses in the internal memory are organized by the form of Hash (HASH) table.In MemoryCache, some property values of main in store cache file, cache table of articles (CacheEntryTable) and MemoryCache, for example chained list (EvicLink) of the real bytes number (Size) of maximum number of byte (MaxBytes), maximum entry number (MaxEntryCount), MemoryCache, actual file number (EntryCount), deletion ancient deed.
DiskCache has then write down cache file and the index information thereof on the disk, and DiskCache organizes the index information of cache file on the disk by the form of HASH table.Index information mainly comprises the size of cache filename, cache file, blocks of files ID etc.In DiskCache, some property values of main in store cache file, cache table of articles (CacheRecordTable) and DiskCache, for example chained list (EvicLink) of the real bytes number (Size) of maximum number of byte (MaxBytes), maximum entry number (MaxFileCount), DiskCache, actual file number (FileCount), deletion ancient deed.
Shown in Figure 2 is the organization chart of cache clauses and subclauses, has described the global tissue form that is used for the cache clauses and subclauses that WEB quickens, and cahe clauses and subclauses figure has described the content of cache clauses and subclauses of the URL correspondence of each file.With reference to Fig. 2, each cache entry record the HTTP head of a URL correspondence, HTTP data, the HTTP data validity information buffer memorys in internal memory such as (Expiretime, LastModifyTime).In the cache clauses and subclauses, index (Key) is that URL is carried out the Key that obtains behind the HASH, can be according to the cache clauses and subclauses of this index search correspondence; FileHash carries out the HASH value that HASH obtains to the cache file, can be used for the security of authenticating documents; Remaining all is the content of stipulating in the existing http protocol, repeats no more here.Can directly use the URL of WEB file file to be carried out buffer memory as index, but the length difference of different URL is bigger, on specific implementation, need to be provided with the length of index very big as index, therefore in embodiments of the present invention, URL is carried out HASH, obtain the unified HASH value of length, URL is obtained the Key of a regular length by HASH.Utilize this HASH value as index then, it is more convenient to make that like this technical scheme of the present invention implements.Have a cache clauses and subclauses HASH table among the MemoryCache, its key value is obtained by HASH by above-described URL, and corresponding value (value) then is the cache clauses and subclauses in the internal memory.Have a cache index information HASH table among the DiskCache, its key value is obtained by HASH by above-described URL, and corresponding value then is the index information of cache file.
Schematic flow sheet when shown in Figure 3 is during WEB quickens browser or other device need obtain the cache file.With reference to Fig. 3, this flow process may further comprise the steps:
Step 101, in the time of need obtaining the WEB file of certain URL, is then obtained by CacheMgr in the WEB accelerator to step 102.CacheMgr searches in MemoryCache according to the HASH value of this URL of calculating, and whether see has corresponding cache clauses and subclauses existence in the internal memory, if having then directly hit execution in step 103; Otherwise, execution in step 104 and subsequent step thereof.
Step 103 reads corresponding file, and further offers corresponding module from the cache clauses and subclauses of MemoryCache, browser etc. for example, and this flow process finishes then.
Step 104 is to step 105, CacheMgr searches in DiskCache according to the HASH value of URL, judge whether that corresponding cache clauses and subclauses exist, if have, then need these cache clauses and subclauses are read and are written to the MemoryCache from DiskCache, so execution in step 106 and subsequent step thereof; Otherwise, need be from network download this document, and create new cache clauses and subclauses and write MemoryCache, then execution in step 109 and subsequent step thereof.
Step 106, because MemoryCache has the restriction of maximum committed memory space and maximum cache entry number, so judge in this step whether MemoryCache is full, promptly cache uses internal memory whether to surpass above-mentioned restriction, if, then execution in step 107 and subsequent step thereof; Otherwise execution in step 108.
Step 107, CacheMgr is about to LastFetchTime some cache clauses and subclauses deletions the earliest, and writes DiskCache some do not have the cache clauses and subclauses of accessed mistake to flush to disk to get at most among the MemoryCache.
Step 108, CacheMgr is according to the index information among the DiskCache, the new cache clauses and subclauses of structure in MemoryCache, file in the clauses and subclauses of cache described in the DiskCache is write in the cache clauses and subclauses of being constructed, and read described file, can further this document be offered corresponding module then, as browser etc., so far this flow process finishes.
Step 109 judges whether MemoryCache is full, if, then execution in step 110 and subsequent step thereof; Otherwise, execution in step 111.
Step 110, CacheMgr is about to LastFetchTime some cache clauses and subclauses deletions the earliest, and writes DiskCache some do not have the cache clauses and subclauses of accessed mistake to flush to disk to get at most among the MemoryCache.
Step 111, according to downloaded files, create corresponding new cache clauses and subclauses at MemoryCache, with the HASH value of URL as the Key in the cache clauses and subclauses, and this document is write the cache clauses and subclauses of new establishment, can further downloaded files be offered corresponding module then, as browser etc., so far this flow process finishes.
Obtain after file finishes at user capture cache, further can for example system's free time, some accessed cache clauses and subclauses deletions will do not had at most among the MemoryCache in due course, and be written among the DiskCache, and upgrade the cache index information of DiskCache.In addition, in above-mentioned steps 107 and 110, also need the cache clauses and subclauses are write disk.Shown in Figure 4 is exactly the schematic flow sheet that the cache clauses and subclauses is write DiskCache.With reference to Fig. 4, this flow process may further comprise the steps:
Step 201 is because DiskCache has maximum to take the restriction of disk space and maximum cache number of files, therefore, CacheMgr judges whether DiskCache is full, be that cache uses whether to have surpassed above-mentioned restriction, if, then execution in step 202 and subsequent step thereof; Otherwise, execution in step 203.
Step 202, from some cache clauses and subclauses that are not updated at most of disk deletion, the cache file that promptly LastModifyTime is the oldest, and the corresponding disk directory entry of deletion.
Step 203 writes DiskCache with the cache clauses and subclauses among the described MemoryCache.
In above-mentioned flow process, CacheMgr need conduct interviews to the cache clauses and subclauses among MemoryCache and the DiskCache.Visit to the cache clauses and subclauses has the read and write dual mode.It is to read information such as the HTTP head of URL correspondence and HTTP file from the cache clauses and subclauses that the mode of reading is visited.The WriteMode visit is that information such as HTTP head and HTTP file are write in the cache clauses and subclauses, in order to subsequent access.
As shown in Figure 5, the embodiment of the invention has further proposed idiographic flow that the cache clauses and subclauses are conducted interviews.In this flow process, need in the cache clauses and subclauses, further write down the current accessed state of oneself, comprise access mode and visit the number of users of oneself to read mode, promptly read number of users.Wherein, access mode can be divided into: read mode, WriteMode and do not have user capture.
With reference to Fig. 5, this flow process may further comprise the steps:
Step 301 reads the current accessed state of cache clauses and subclauses from the cache clauses and subclauses, if WriteMode, the then current visit of refusal can begin visit next time then, and execution in step 301 once more; If read mode, then execution in step 302 and subsequent step thereof; If there is not user capture, then execution in step 307 and subsequent step thereof.
Step 302, judging current is to read or to write, if write, the then current visit of refusal can begin visit next time then, execution in step 301 once more; If read mode, then execution in step 303 and subsequent step thereof.
Step 303 is visited the cache clauses and subclauses to read mode, and the number of users of reading in the current state is added 1.
Step 304 reads after the end of file in the cache clauses and subclauses, will read number of users and subtract 1.
Step 305 is judged and current read number of users and whether equal 0, if, then execution in step 306 and subsequent step thereof, otherwise can begin visit next time, execution in step 301 once more.
Step 306, record current state can begin visit next time then for there not being user capture, and execution in step 301 once more.
Step 307, judging current is to read or to write, if read, then execution in step 303 and subsequent step thereof; If write, then execution in step 308 and subsequent step thereof.
Step 308 with WriteMode visit cache clauses and subclauses, writes file and relevant information.
Step 309 writes after the end, and the current state of record cache clauses and subclauses is not for there being user capture.Then, can carry out visit next time, execution in step 301 once more.
In flow process shown in Figure 5, write or read and can be divided into following two kinds of situations according to what will carry out:
Situation 1: in the cache clauses and subclauses, write file.Obtain the current accessed state of cache clauses and subclauses,, forbid writing when the current accessed state of cache clauses and subclauses is WriteModes or when reading mode; When the current accessed state of cache clauses and subclauses is when not having user capture, in these cache clauses and subclauses, write file, and the current state of record cache clauses and subclauses is a WriteMode; After writing end, the current state of record cache is not for there being user capture.
Situation 2: read the file in the cache clauses and subclauses.Obtain the current accessed state of cache clauses and subclauses, when the current accessed state of cache clauses and subclauses is WriteMode, forbid reading; When the current accessed state of cache clauses and subclauses is when reading mode or not having user capture, read the file in the cache clauses and subclauses, and the number of users of reading of cache clauses and subclauses is added 1; After reading end, when the reading number of users and be 0 of cache clauses and subclauses, the current state of record cache clauses and subclauses is not for there being user capture.
In the cache clauses and subclauses in reading internal memory, can further authenticate its legitimacy.The source of HTTP head and HTTP file has two kinds in the cache clauses and subclauses: obtain the cache clauses and subclauses from disk, perhaps download in the users from networks and write the cache clauses and subclauses with WriteMode.Wherein, from network, download to be divided into and be that the HTTP mode is downloaded fully and not exclusively be that the HTTP mode is downloaded, and not exclusively be that the HTTP mode can being divided into again of downloading is that the P2P mode is downloaded and by the P2P mode with the HTTP mode is collaborative downloads fully.
As shown in Figure 6 be the schematic flow sheet that the cache clauses and subclauses is carried out legitimacy authentication.With reference to Fig. 6, this flow process comprises:
Step 401 judges that these cache clauses and subclauses obtain from disk, still from network, downloads, if from disk, obtain, and then execution in step 402 and subsequent step thereof; If from network, download, then execution in step 403 and subsequent step thereof.
Step 402 when reading the cache file from disk, when reading HTTP head and HTTP file, reads the HASH value V1 of cache file from disk.Then, HTTP head and HTTP file are a HASH and calculate a new HASH value V2.Judge whether V1 equals V2, if equal, think that then the cache file is legal, otherwise think that the cache file is illegal, process ends.
Step 403, judge institute download the file that writes the cache clauses and subclauses whether be complete by the HTTP mode from the WEB downloaded, if, think that then it is legal, and HTTP head and HTTP file are a HASH calculate a HASH value, in order to follow-up use, process ends then; Otherwise, execution in step 404 and subsequent step thereof.
Step 404, the HTTP head of this situation and HTTP file have some or all of the download in the P2P mode to obtain, and obtain the HASH value V3 that also can obtain this cache file in the other side's buffer memory simultaneously by the P2P mode in download.Then, the HTTP head and the HTTP file of this locality are done a HASH calculating, obtain a new HASH value V4.Judge whether V3 equals V4, if equal, think that then the cache file is legal, otherwise think that the cache file is illegal, and process ends.
In the WEB accelerator, sudden large amount of small documents is write affiliation and is caused the CPU expense of system higher, if in visit cache clauses and subclauses, contents such as HTTP head and file are write disk file, more can reduce user capture cache efficient, thereby influence the effect that WEB quickens.The embodiment of the invention is in order to improve the efficient of user capture cache, thereby further improves the effect that WEB quickens, and proposed the method that cache file induction disk writes and big file merges.
Cache file induction disk writes and is meant, just the cache file is written in the internal memory of cache clauses and subclauses in the time of user capture cache, after visit is finished the cache clauses and subclauses of correspondence is submitted to a cache and refreshes in the chained list and go.CacheMgr provides one independently to refresh thread, and according to the busy situation of system, the content that will refresh the cache clauses and subclauses in the chained list in system when not busy is write disk and got on, and upgrades the index information of DiskCache correspondence simultaneously.
It is by a plurality of small documents being merged into a big file that big file merges, thereby has reduced the number of times that File Open is closed in the cache clauses and subclauses, reduces system overhead, further improves the speed of visit cache.
Here, being small documents with size less than the document definition of threshold value, and being big file with size greater than the document definition of threshold value, might as well be that 32KB is the example explanation with the threshold value.Simultaneously, according to the file size classification, for example: less than the file of 1KB, the file of 1KB-2KB, the file of 2KB-4BK, the file of 4KB-8KB, the file of 8KB-16KB, the file of 16K-32K.
The small documents of identical category is merged into a big file, and the file that surpasses 32KB for size does not then need to merge, and the largest amount that can set big file is 320KB.The head of each big file has index information, writes down the operating position of oneself.In the present embodiment, big top of file is not included among the described 320KB.The big file that is merged into the small documents less than 1K is the example explanation below, and other situations can be analogized, and repeat no more here.
With reference to Fig. 7, whole big file is divided into two parts: the document entity part that big top of file and small documents piece are formed.Big top of file has write down following information: the number of the small documents that is not used (UnusedCounts), the small documents piece ID (FirstUnusedID) that first is not used, unsigned int (unsignedshort) array of indexes (UnusedIDs) that size is 320Byte, each member of array is corresponding to a small documents piece, if this small documents piece also is not used, its content is the ID of the next small documents piece that is not used so.The file that can to deposit a size in each small documents piece be 1KB or less than the file of 1KB.
With reference to Fig. 8, to be no more than the process of file of 1KB as follows when the user writes a size:
Step 501 finds one to also have the big file of idle small documents piece or create a big file, and the big or small classification of small documents is identical with the small documents that will write in the big file that is found.
Step 502 reads FirstUnuseID, and the small documents that write is write first idle small documents piece.
Step 503 is upgraded big top of file, that is:
FirstUnusedID=UnusedIDs[FirstUnusedID]
UnusedCounts=UnusedCounts+1
With reference to Fig. 9, the process of deleting the small documents that a size is no more than 1KB as the user is as follows:
Step 601 finds corresponding big file and small documents piece ID:IDi according to the index information of DiskCache, then the small documents that will delete of deletion.
Step 602 is upgraded big top of file, that is:
UnusedIDs[IDi]=FirstUnusedID
FirstUnusedID=IDi
UnusedCounts=UnusedCounts-1
Step 603 when the UnusedCounts of this moment equals 320, illustrates that this big file has not had small documents, deletes this big file so.
The above only is preferred embodiment of the present invention, and is in order to restriction the present invention, within the spirit and principles in the present invention not all, any modification of being done, is equal to replacement, improvement etc., all should be included within protection scope of the present invention.

Claims (11)

1, a kind of method for caching and processing is characterized in that, this method comprises:
A. when needs when network obtains file, search in the memory cache module according to index, if find the corresponding cache clauses and subclauses, then read file wherein, otherwise execution in step B; Described index is that the uniform resource position mark URL of Web file is carried out the unified HASH value of length that obtains behind the Hash HASH;
B. search in the disk buffering module according to index, if find the corresponding cache clauses and subclauses, then in the memory cache module, construct new clauses and subclauses, with the file in the corresponding cache clauses and subclauses in the described disk buffering module write construct in the cache entries, and read described file, otherwise execution in step C;
C. from network, download and obtain described file, the new cache entries of structure in the memory cache module, and in this cache entries, write described file;
This method further comprises: the cache entries in the memory cache module is write the disk buffering module;
When writing size less than the small documents of setting threshold, with its write to comprise the big or small classification of small documents identical with the small documents that will write and have the big file in enough spaces or it is write the big file of establishment;
When the described small documents of deletion, this small documents of deletion from the big file at its place, and when not having small documents in the described big file, delete described big file;
When reading described small documents, from the big file at its place, read this small documents.
2, method according to claim 1, it is characterized in that, in the memory cache module, further comprise the step of judging that the memory cache module is whether full among step B and the step C before the new cache entries of structure, if, then delete the cache entries that does not read at most in the memory cache module, carry out structure then; Otherwise, directly carry out structure.
3, method according to claim 2 is characterized in that, in the cache entries it is being write the disk buffering module described in the deletion memory cache module.
4, method according to claim 1 is characterized in that, describedly cache entries in the memory cache module is write the disk buffering module is: will not have accessed cache entries to write the disk buffering module in the memory cache module at most.
5, according to claim 3 or 4 described methods, it is characterized in that, before writing the disk buffering module, further comprise the step of judging that the disk buffering module is whether full, if, then delete the cache entries that is not modified at most in the disk buffering module, carry out writing then; Write otherwise directly carry out.
6, method according to claim 1 is characterized in that, the described step that writes file in cache entries comprises:
Obtain the current accessed state of cache entries,, forbid writing when the current accessed state of cache entries is a WriteMode or when reading mode;
When the current accessed state of cache entries is when not having user capture, in this cache entries, write file, and the current state of record buffer memory clauses and subclauses is a WriteMode; After writing end, the current state of record buffer memory is not for there being user capture.
7, method according to claim 1 is characterized in that, the step that reads the file in the cache entries comprises:
Obtain the current accessed state of cache entries, when the current accessed state of cache entries is WriteMode, forbid reading;
When the current accessed state of cache entries is when reading mode or not having user capture, read the file in the cache entries, and the number of users of reading of cache entries is added 1; After reading end, when the reading number of users and be 0 of cache entries, the current state of record buffer memory clauses and subclauses is not for there being user capture.
8, method according to claim 1 is characterized in that, this method further comprises:
When obtaining cache entries from the disk buffering module, read the wherein HASH value of file, and calculate the HASH value of the file that is read, if two HASH values are identical, it is legal then authenticating described file, is illegal otherwise authenticate described file; And/or,
Authentication is legal by HTTP mode downloaded files; And/or,
Some or all of during by point-to-point mode file in download, download corresponding HASH value, and the HASH value of calculating institute downloaded files, if two HASH values are identical, it is legal then authenticating described file, is illegal otherwise authenticate described file.
9, method according to claim 1, it is characterized in that, this method further comprises: if in the access cache clauses and subclauses cache entries is write the disk buffering module, then with the cache entries write memory, the cache entries with write memory after to be visited the finishing is submitted in the cache flush chained list; Provide one independently to refresh thread, the cache entries in the described cache flush chained list is write in the described disk buffering module during free time in system, and upgraded the index information of disk buffering module correspondence simultaneously.
10, a kind of caching process system is characterized in that this system comprises memory cache module, disk buffering module and caching management module, wherein:
The memory cache module is used in internal memory memory buffers clauses and subclauses;
The disk buffering module is used in disk memory buffers clauses and subclauses;
Caching management module, be used for when needs when network obtains file, search in the memory cache module according to index, if find the corresponding cache clauses and subclauses then read wherein file, otherwise in the disk buffering module, search according to index, if find the corresponding cache clauses and subclauses, then in the memory cache module, construct new cache entries, file in the corresponding cache clauses and subclauses in the described disk buffering module is write in the cache entries of being constructed, and read described file, otherwise from network, download and obtain described file, the new cache entries of structure in the memory cache module, and in this cache entries, write described file; Cache entries in the memory cache module is write the disk buffering module; When writing size less than the small documents of setting threshold, with its write to comprise the big or small classification of small documents identical with the small documents that will write and have the big file in enough spaces or it is write the big file of establishment; When the described small documents of deletion, this small documents of deletion from the big file at its place, and when not having small documents in the described big file, delete described big file; When reading described small documents, from the big file at its place, read this small documents;
Described index is that the URL to the Web file carries out the unified HASH value of length that obtains behind the HASH.
11, system as claimed in claim 10, it is characterized in that, described caching management module is further used for, if in the access cache clauses and subclauses, cache entries write the disk buffering module, then with the cache entries write memory, the cache entries with write memory after to be visited the finishing is submitted in the cache flush chained list; Provide one independently to refresh thread, the cache entries in the described cache flush chained list is write in the described disk buffering module during free time in system, and upgraded the index information of disk buffering module correspondence simultaneously.
CNB2006101671808A 2006-12-26 2006-12-26 Method and system for processing buffer Active CN100530186C (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CNB2006101671808A CN100530186C (en) 2006-12-26 2006-12-26 Method and system for processing buffer

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CNB2006101671808A CN100530186C (en) 2006-12-26 2006-12-26 Method and system for processing buffer

Publications (2)

Publication Number Publication Date
CN101075241A CN101075241A (en) 2007-11-21
CN100530186C true CN100530186C (en) 2009-08-19

Family

ID=38976294

Family Applications (1)

Application Number Title Priority Date Filing Date
CNB2006101671808A Active CN100530186C (en) 2006-12-26 2006-12-26 Method and system for processing buffer

Country Status (1)

Country Link
CN (1) CN100530186C (en)

Families Citing this family (52)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101576855B (en) * 2009-06-19 2011-05-04 深圳市科陆电子科技股份有限公司 Data storing system and method based on cache
CN101710327A (en) * 2009-12-04 2010-05-19 深圳创维数字技术股份有限公司 Method for caching local temporary files into embedded browser
CN102117309B (en) * 2010-01-06 2013-04-17 卓望数码技术(深圳)有限公司 Data caching system and data query method
CN102253941A (en) * 2010-05-21 2011-11-23 卓望数码技术(深圳)有限公司 Cache updating method and cache updating device
CN101877010A (en) * 2010-06-12 2010-11-03 中兴通讯股份有限公司 Distributed file system and associated storage method thereof
CN101908077B (en) * 2010-08-27 2012-11-21 华中科技大学 Duplicated data deleting method applicable to cloud backup
CN102447688A (en) * 2010-10-15 2012-05-09 盛绩信息技术(上海)有限公司 Webpage game resource accelerator and acceleration method
CN102479250A (en) * 2010-11-26 2012-05-30 中国科学院声学研究所 Disk caching method for embedded browser
CN102054031A (en) * 2010-12-22 2011-05-11 上海锐起信息技术有限公司 Method for realizing network disc local write cache control in disk-free system
CN102622347A (en) * 2011-01-26 2012-08-01 腾讯科技(深圳)有限公司 Image management method and movable terminal
CN102737068B (en) * 2011-04-15 2018-06-05 北京百度网讯科技有限公司 A kind of method and apparatus for being used to carry out retrieval data cache management
CN102298527B (en) * 2011-06-21 2017-03-15 中兴通讯股份有限公司 A kind of method of CPE parallel upgrades version and realize system
CN102368258B (en) * 2011-09-30 2014-11-26 广州市动景计算机科技有限公司 Webpage page caching management method and system
CN103106216A (en) * 2011-11-15 2013-05-15 腾讯科技(深圳)有限公司 Method and device of web page access
CN102411631A (en) * 2011-12-22 2012-04-11 中国科学院声学研究所 Disk cache method of embedded browser
CN103246611B (en) * 2012-02-07 2016-01-20 腾讯科技(深圳)有限公司 A kind of document handling method and system
CN102609364A (en) * 2012-02-14 2012-07-25 华中科技大学 Cache exchange method
CN103294718A (en) * 2012-02-29 2013-09-11 腾讯科技(深圳)有限公司 Method and device for web page cache management
CN102638570A (en) * 2012-03-15 2012-08-15 中兴通讯股份有限公司 Embedded network agent system, terminal equipment and embedded network agent method
CN103369284A (en) * 2012-03-31 2013-10-23 浙江大华技术股份有限公司 Audio video data prerecording method and device
CN102663096B (en) * 2012-04-11 2015-12-16 北京像素软件科技股份有限公司 A kind of method reading data based on Data cache technology
CN103514203A (en) * 2012-06-27 2014-01-15 腾讯科技(深圳)有限公司 Method and system for browsing webpage in reading mode
CN103077198B (en) * 2012-12-26 2016-09-21 中标软件有限公司 A kind of operating system and file cache localization method thereof
CN103051977A (en) * 2013-01-11 2013-04-17 乐视网信息技术(北京)股份有限公司 Method for processing p2p (peer-to-peer) cache data
CN103970585B (en) * 2013-01-31 2017-10-27 新华三技术有限公司 Create the method and device of virtual machine
CN103279562B (en) * 2013-06-09 2017-10-10 网易(杭州)网络有限公司 A kind of method, device and database storage system for database L2 cache
CN104424116B (en) * 2013-08-19 2017-07-07 中国科学院声学研究所 A kind of method and system of built-in browser disk buffering
CN104423784A (en) * 2013-09-04 2015-03-18 腾讯科技(深圳)有限公司 Page displaying method and page displaying device
CN104572643B (en) * 2013-10-10 2018-06-19 北大方正集团有限公司 Searching method and search engine
CN103873562A (en) * 2014-02-27 2014-06-18 车智互联(北京)科技有限公司 Cache method and cache system
CN104298562A (en) * 2014-10-10 2015-01-21 重庆邮电大学 Resource management method and system for game development
CN104331428B (en) * 2014-10-20 2017-07-04 暨南大学 The storage of a kind of small documents and big file and access method
CN104363182B (en) * 2014-11-04 2018-07-31 浪潮(北京)电子信息产业有限公司 A kind of load-balancing method and system based on bilayer caching
CN104572194B (en) * 2014-12-30 2019-03-08 北京奇虎科技有限公司 A kind of module loading method and apparatus
CN104794068B (en) * 2015-02-04 2018-05-22 深圳神州数码云科数据技术有限公司 A kind of method and device of flush cachings
CN106294206B (en) * 2015-05-21 2022-04-29 广州华多网络科技有限公司 Cache data processing method and device
CN104978154A (en) * 2015-07-27 2015-10-14 武汉天喻信息产业股份有限公司 Cache operation-based flash rapid read-write method and system
CN105718291B (en) * 2016-01-25 2020-07-07 山东蓝创网络技术有限公司 Multilevel cache acceleration method for mixed desktop application
CN106372109A (en) * 2016-08-19 2017-02-01 中国银联股份有限公司 Internet resource file caching method and apparatus
CN106570134A (en) * 2016-10-27 2017-04-19 西安合众思壮导航技术有限公司 File organization method and apparatus
CN106776368B (en) * 2016-11-29 2021-02-05 苏州浪潮智能科技有限公司 Cache management method, device and system during data reading
CN108182187A (en) * 2016-12-08 2018-06-19 腾讯科技(深圳)有限公司 Data cache method and device
CN106599247B (en) * 2016-12-19 2020-04-17 北京奇虎科技有限公司 Method and device for merging data files in LSM-tree structure
CN107463508A (en) * 2017-07-19 2017-12-12 北京微影时代科技有限公司 Script caching sweep-out method, device, electronic equipment and storage medium
CN110597452A (en) * 2018-06-13 2019-12-20 中国移动通信有限公司研究院 Data processing method and device of storage system, storage server and storage medium
CN108846141B (en) * 2018-07-11 2022-05-24 中国建设银行股份有限公司 Offline cache loading method and device
CN108965910B (en) * 2018-08-09 2022-05-10 网宿科技股份有限公司 Video caching method and player
CN109815425A (en) * 2018-12-14 2019-05-28 平安科技(深圳)有限公司 Caching data processing method, device, computer equipment and storage medium
CN109361714B (en) * 2018-12-18 2021-11-16 中国移动通信集团江苏有限公司 User login authentication method, device, equipment and computer storage medium
CN110109958B (en) * 2019-04-19 2024-09-03 深圳前海微众银行股份有限公司 Cache processing method, device, equipment and computer readable storage medium
CN112734982A (en) * 2021-01-15 2021-04-30 北京小马慧行科技有限公司 Storage method and system for unmanned vehicle driving behavior data
CN116599973B (en) * 2023-07-11 2023-09-26 天津卓朗昆仑云软件技术有限公司 Data transmission method and device based on BT service acceleration component

Also Published As

Publication number Publication date
CN101075241A (en) 2007-11-21

Similar Documents

Publication Publication Date Title
CN100530186C (en) Method and system for processing buffer
US10958752B2 (en) Providing access to managed content
US9509748B2 (en) Efficient storage and retrieval of resources for rendering structured documents
CN109240946B (en) Multi-level caching method of data and terminal equipment
US8650164B2 (en) Efficient storage and retrieval for large number of data objects
CN100456311C (en) System and method for actualizing content-based file system security
CN103179185B (en) Method and system for creating files in cache of distributed file system client
US10061852B1 (en) Transparent proxy tunnel caching for database access
US6952730B1 (en) System and method for efficient filtering of data set addresses in a web crawler
CN102985921B (en) There is the client terminal device high speed caching electronic document resources of e-sourcing data base
CN108920600B (en) Distributed file system metadata prefetching method based on data relevance
CN104679898A (en) Big data access method
CN105183839A (en) Hadoop-based storage optimizing method for small file hierachical indexing
CN104778270A (en) Storage method for multiple files
CN103902660B (en) System and method for prefetching file layout through readdir++ in cluster file system
US20140259100A1 (en) Network security identification method, security detection server, and client and system therefor
US20130318211A1 (en) Asset streaming and delivery
CN104021137B (en) A kind of client based on catalogue mandate is locally opened and closed the method and system of file
US7249219B1 (en) Method and apparatus to improve buffer cache hit rate
CN105701233B (en) A method of optimization server buffer management
CN103631914A (en) Offline browsing method and system of massive internet content
US9178931B2 (en) Method and system for accessing data by a client from a server
CN107015978A (en) A kind of web page resources processing method and device
US11586353B2 (en) Optimized access to high-speed storage device
CN109002495A (en) Date storage method and device

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
C14 Grant of patent or utility model
GR01 Patent grant
TR01 Transfer of patent right

Effective date of registration: 20230921

Address after: 518057 Tencent Building, No. 1 High-tech Zone, Nanshan District, Shenzhen City, Guangdong Province, 35 floors

Patentee after: TENCENT TECHNOLOGY (SHENZHEN) Co.,Ltd.

Patentee after: TENCENT CLOUD COMPUTING (BEIJING) Co.,Ltd.

Address before: 2, 518044, East 403 room, SEG science and Technology Park, Zhenxing Road, Shenzhen, Guangdong, Futian District

Patentee before: TENCENT TECHNOLOGY (SHENZHEN) Co.,Ltd.

TR01 Transfer of patent right