CN101178693A - Data cache method and system - Google Patents
Data cache method and system Download PDFInfo
- Publication number
- CN101178693A CN101178693A CNA200710195981XA CN200710195981A CN101178693A CN 101178693 A CN101178693 A CN 101178693A CN A200710195981X A CNA200710195981X A CN A200710195981XA CN 200710195981 A CN200710195981 A CN 200710195981A CN 101178693 A CN101178693 A CN 101178693A
- Authority
- CN
- China
- Prior art keywords
- data
- piece
- row
- blocks
- reference position
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 61
- 230000015654 memory Effects 0.000 claims abstract description 64
- 230000003139 buffering effect Effects 0.000 claims abstract description 5
- 238000012545 processing Methods 0.000 abstract description 5
- 238000000638 solvent extraction Methods 0.000 abstract 2
- 238000004422 calculation algorithm Methods 0.000 description 22
- 230000008569 process Effects 0.000 description 17
- 238000010586 diagram Methods 0.000 description 13
- 238000005516 engineering process Methods 0.000 description 5
- 238000004364 calculation method Methods 0.000 description 3
- 238000013500 data storage Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 230000004044 response Effects 0.000 description 3
- 238000007514 turning Methods 0.000 description 3
- 238000002679 ablation Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000000151 deposition Methods 0.000 description 2
- 230000006399 behavior Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000001934 delay Effects 0.000 description 1
- 238000009795 derivation Methods 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 239000012467 final product Substances 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000007639 printing Methods 0.000 description 1
- 239000000047 product Substances 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data buffering method and a system to solve the problems of the mutual collision of the memory consumption and the visiting speed of the existing report forms buffering method. The method comprises that: the invention divides the storing space into an index district and a data district; the data is partitioned; each block of data is sequentially written into the data district of the storing space and the initial position of each block is sequentially written into the index district according to the partitioning result; and the data is read according to the partitioning result and the initial position of each block. The buffering technique reduces the occupation of the memory, because of reading the data from the fixed disk every time, can greatly save the use of the memory space and elevate concurrent processing ability of the system under the status of multiple concurrent and big data quantity; besides, the invention greatly improves the efficiency of the document reading by the index method to read the report forms from fixed disk. Therefore, the method of the invention gets a balance between the memory space and the visiting speed.
Description
Technical field
The present invention relates to the metadata cache technology, particularly relate to a kind of data cache method and system.
Background technology
Form is a kind of data representation format with row-column configuration, expression be the representing of business datum on the time point.With reference to shown in Figure 1, be a report form embodiment figure, what the figure shows is a user message table, comprises 3 row, 4 row, totally 12 cells.In a lot of application systems, the result of system's operation shows by form.
In actual applications, after system generated form, the user still had some processing of form hereto, as page turning, printing, derivation PDF, Excel formatted file etc.In this process, if system regenerates form at every turn, then can increase the load of report server, database server, and increase user's stand-by period; Generate for twice in the process of form simultaneously, the data in the database may change, will cause the user to see like this by browser and the form content that prints inconsistent.Therefore, efficient, a perfect report server must have the ability of buffer memory report data.
For the buffer memory of form, two kinds of typical methods are arranged at present: a kind of method is the complete write memory of report data, then by " least referenced recently " or " weighting " scheduling algorithm, the cache list of management maintenance form; Another kind method is the complete exterior storage medium such as hard disk that writes of report data, before to the report data visit, loads report data from hard disk at every turn.
Described two kinds of methods provide different caching report mechanism for application system, but all have shortcoming: first method is because the report data write memory, for the application that concurrency is big, the report data amount is big, can consume the internal memories of application servers in a large number, cause server efficient to reduce, even occur that internal memory overflows, system delays phenomenons such as machine; Though and second method has been avoided a large amount of consumption of Installed System Memory, but the response time that each access hard disk reads whole form content, far above the running time of reading from internal memory, the read-write of fixed disk file simultaneously can take a large amount of CPU time, thereby influences the performance of total system.
Summary of the invention
Technical matters to be solved by this invention provides a kind of data cache method and system, to solve existing caching report method afoul problem on memory consumption and access speed.
For solving the problems of the technologies described above,, the invention discloses following technical scheme according to specific embodiment provided by the invention:
A kind of data cache method comprises:
External memory space is divided into index area and data field;
With deblocking;
According to the deblocking result, every blocks of data is write the data field of described external memory space successively, the reference position of every blocks of data is write successively the index area of described external memory space;
According to the reference position of deblocking result and every blocks of data, reading of data.
Wherein, described data have row-column configuration, carry out piecemeal according to row or row during deblocking.
A kind of preferred data cache method comprises:
External memory space is divided into index area and data field, and described data field is divided into piece index area and blocks of data district to data block again;
With deblocking;
According to the deblocking result, every blocks of data is write the data field of described external memory space successively, the reference position of every blocks of data is write successively the index area of described external memory space; When writing every blocks of data, each row of data is write the blocks of data district of this data block successively, the reference position of every row is write successively the piece index area of this data block;
According to the reference position of deblocking result, every blocks of data and the reference position of each row of data, reading of data.
Wherein, carry out deblocking according to following steps: preset largest block number and every minimum line number; If all line numbers of data are greater than the product of largest block number and every minimum line number, then every line number equals every minimum line number and multiply by 2; Otherwise every line number equals every minimum line number, and the data block number equals (all line numbers+every line number-1) every line number of ÷.
Wherein, the result writes data according to following steps according to piecemeal: count computation index district size according to piece, obtain the data field reference position; According to line number computing block index area size in the piece, obtain blocks of data district reference position; Write current line, write down the current line position, and obtain the reference position of next line; After writing a blocks of data, the reference position of every row in this piece is write the piece index area of this piece; After writing all data blocks, every reference position is write the index area of storage space.
Preferably, described method also comprises: the line number that every blocks of data is comprised is the write-in block index area also.
Wherein, according to the piecemeal result according to the following steps reading of data: according to piece number and every line number, obtain asking row place piece and the sequence number in piece; According to request row place piece, inquire about the index area of storage space, obtain the reference position of this piece; According to the sequence number of request row in piece, continue the piece index area of this piece of inquiry, obtain the reference position of this request row; Read line data from the reference position of this request row.
Preferably, described method also comprises: the data line that buffer memory reads, when cell in this row is read in request, directly read from buffer memory.
Wherein, described data have row-column configuration, and described according to going the data of carrying out piecemeal, by the ranks coordinate transform, replaceable one-tenth carries out the data of piecemeal according to row.
Preferably, the each execution when reading or writing data all carried out a deblocking operation and obtained the piecemeal result.
A kind of data buffering system comprises:
Storage unit is used for external memory space is divided into index area and data field, and wherein data area stores is divided into the data of piece, and the reference position of every blocks of data is stored in the index area;
The deblocking unit is used for deblocking;
Write operation unit is used for according to the deblocking result, and every blocks of data is write the data field of described external memory space successively, the reference position of every blocks of data is write successively the index area of described external memory space;
The read operation unit is used for the reference position according to deblocking result and every blocks of data, reading of data.
Wherein, described data have row-column configuration, and described deblocking unit carries out piecemeal according to row or row.
Preferably, in the described storage unit, the data field is divided into piece index area and blocks of data district to data block again, and wherein the blocks of data district of every blocks of data stores all line data of this piece, and the reference position of every row in this piece is stored in the piece index area.
Preferably, the line number that this blocks of data comprises is also stored in the piece index area of described every blocks of data.
Preferably, described system also comprises: buffer unit, be used for the data line that buffer memory reads, and when cell in this row is read in request, directly from buffer unit, read.
According to specific embodiment provided by the invention, the invention discloses following technique effect:
The embodiment of the invention provides a kind of data cache method, at the report data with row-column configuration, the report data piecemeal is stored in the file.Hard drive space with document form storage data is divided into index area and data field, data area stores block data wherein, and the reference position of every blocks of data is stored in the index area, by write operation report data is write file.Like this, when reading report data, just can directly navigate to the piece at place, in piece, find needed data fast according to the index area.
Above-mentioned caching technology is only preserved report name (distinguishing different files by file name in hard disk) in internal memory, and report data is left in the exterior storage mediums such as hard disk, loads corresponding data according to report name from hard disk when reading at every turn.The present invention is for the visit of form, owing at every turn from the hard disk reading of data, reduced taking of internal memory, under the situation of how concurrent, big data quantity, the use that can save memory headroom in a large number improves the concurrent processing ability of system; Simultaneously, read report file from hard disk, greatly improved the efficient that file reads by this indexed mode.Therefore, the method for the invention has obtained a balance between memory headroom and access speed.
Preferably, the data field of file is divided into piece index area and blocks of data district according to data block again, and wherein the blocks of data district of every blocks of data stores all line data of this piece, and the reference position of every row in this piece is stored in the piece index area.Like this, when reading report data, just can locate fast, promptly find the piece at request msg place earlier in the file index district, find the reference position of request msg then in the piece index area of this piece, and then read desired data by two layer indexs.
Preferably, in actual applications, the visit of corresponding report data can once be visited usually with each grid in the delegation, thus in internal memory buffer memory the information of delegation.When obtaining visiting the request of a grid, can check whether be certain grid of cache lines,, and do not need from fixed disk file, to have read earlier like this if then can directly use cache lines.
Description of drawings
Fig. 1 is report form embodiment figure of the prior art;
Fig. 2 is the comparison diagram that report data writes the file front and back in the embodiment of the invention;
Fig. 3 is the data storage synoptic diagram after Fig. 2 is improved;
Fig. 4 is the process flow diagram of report data block algorithm in the embodiment of the invention;
Fig. 5 is the process flow diagram that form writes file in the embodiment of the invention;
Fig. 6 is the process flow diagram that reads form in the embodiment of the invention from file;
Fig. 7 is the structured flowchart of the described caching report of embodiment of the invention system.
Embodiment
For above-mentioned purpose of the present invention, feature and advantage can be become apparent more, the present invention is further detailed explanation below in conjunction with the drawings and specific embodiments.
Can not on memory consumption and access speed, meet the requirements of problem simultaneously at caching report mode of the prior art, the embodiment of the invention provides a kind of report data caching method, to writing the report data piecemeal of file, and write down every position hereof, when reading report data, just can directly navigate to the piece at place like this, in piece, find needed data fast.This method is primarily aimed at the data that form etc. has row-column configuration (can with reference to example shown in Figure 1), is that the data characteristic according to the matrix representation that can form by ranks designs.Therefore, the present invention also has the data of row-column configuration applicable to other except that being used for report data.
For the form of big data quantity, normally vertically (OK) or horizontal (row) expansion, the situation that seldom has both direction all to expand is in a large number cut apart so the piecemeal of report data has adopted by row (vertically expansion) or has been listed as (extending transversely).Modal is several thousand even several ten thousand row, the situation of every row tens row, therefore the explanation below all with vertical expansion, be example by the row piecemeal.Corresponding extending transversely, can realize with identical way by the displacement of transverse and longitudinal coordinate, be not described in detail in this.
The basis that the present invention realizes is that the storage mode of report data in file system (being external memory spaces such as hard disk) improved, be about to file system and be divided into index area and data field two parts, wherein block data is deposited in the data field, and the reference position of every blocks of data is deposited in the index area.
With reference to Fig. 2, be the comparison diagram that report data writes the file front and back.
Among the figure, the synoptic diagram in left side represents that report data is divided into polylith according to line number before depositing file system in, and the synoptic diagram on right side is represented the storage mode of report data after writing file system of piecemeal.In the synoptic diagram on right side, report data is that unit is written into file with the piece, the file index district store successively the 1st, the 2nd ..., the N blocks of data reference position, the file data district deposit successively the 1st, the 2nd ..., the N blocks of data, can find any blocks of data hereof according to the data block reference position in the index area.In the present embodiment, for ease of the operations such as read-write of file, the file top is set to the index area, is only the data field then.
Preferably, present embodiment also provides another kind of document storage mode, further the data field of file is divided, and every blocks of data is divided into piece index area and blocks of data district again, wherein the blocks of data district stores all line data of this piece, and the reference position of each row of data in this piece is stored in the piece index area.With reference to shown in Figure 3, be the data storage synoptic diagram after Fig. 2 is improved.Among the figure, the top of file is the index area, in the data field of file, the piece index area of every blocks of data stored successively the 1st the row, the 2nd the row ..., the N line data reference position, every blocks of data district deposited successively the 1st the row, the 2nd the row ..., the N line data, can in data block, find any data line according to the data line reference position in the piece index area.This two-layer indexed mode, can realize reading of report data fast, when searching certain delegation report data, just can locate fast by described two layer indexs, promptly find the piece at this line data place earlier in the file index district, find the reference position of this line data then in the piece index area of this piece, and then find this row reading of data.
Based on above data storage method, the described data cache method of the embodiment of the invention mainly comprises three partial contents, is respectively report data piecemeal, write data and read data, describes in detail respectively below.
(1) report data piecemeal
Data are carried out piecemeal handle several different methods is arranged, because, can formulate the piecemeal result that different block algorithms obtains meeting application demand according to the difference of practical application.Therefore, the present invention does not limit and adopts which kind of block algorithm.
With reference to Fig. 4, be the process flow diagram of report data block algorithm in the embodiment of the invention.The block algorithm that present embodiment adopts is as follows:
Form line number＞largest block number * every minimum line number;
If set up, then execution in step 403; Otherwise, execution in step 404.
Every line number=every minimum line number * 2;
Piece number=every line number of line number ÷;
Wherein, parameter " 2 " can be adjusted according to practical application.
Every line number=every minimum line number;
Piece number=(line number+every line number-1) every line number of ÷.
Illustrate, for a form with 100 line data:
1, by analytical statement line number scope, initialization largest block number=1000, every line number=every minimum line number=32;
2, because form line number 100＜1000 * 32;
So 3 piece numbers=(100+32-1)/32=4.
By on obtain form piecemeal result: the piece number is 4, and every line number is (32,32,32,4).
If the form line number is a lot, for example 1000000, then 1000000＞1000 * 32, every line number has not been " an every minimum line number " 32 just.
The piecemeal result who utilizes block algorithm to obtain comprises piece number and every line number, all can use in the write operation of file and read operation process.When report data is write file, need cut apart report data according to the result that block algorithm is carried out, set up index; When from file, reading report data, need obtain form piece number according to the block algorithm result equally, and then be expert at according to the cell of request and to obtain this row and be stored in which piece, be arranged in piece which is capable.Therefore, block algorithm can be described as basic algorithm of the present invention.
And, in theory, only need when write data, carry out block algorithm, directly obtain the piecemeal result when reading and get final product.But in the practical application, the block algorithm result also needs to take Installed System Memory, and block algorithm itself is than more efficient, and execution speed is very fast.So for save memory headroom as far as possible, present embodiment is all carried out one time block algorithm when writing with reading of data, and can guarantee that twice piecemeal comes to the same thing.
(2) write data
The process of write data is that report data is write file, writes effect such as Fig. 2 or shown in Figure 3 behind the file.To be that example describes below with two layer indexs shown in Figure 3.
With reference to Fig. 5, be the process flow diagram that form writes file in the embodiment of the invention.
Step 501, the result who utilizes block algorithm to carry out counts calculation document index area size according to piece, promptly preserves the space that the piece index needs in the calculation document, obtains the reference position in file data district;
Step 502 judges whether all data blocks all write file, if not, and execution in step 503; If, execution in step 508;
Step 503 is obtained data block to be written;
Step 504 according to line number computing block index area size in the piece, is promptly preserved the space that line index needs in the calculation document data field, obtain blocks of data district reference position, and file pointer is moved on to this position;
Step 505 judges whether row all in this piece all writes file, if not, and execution in step 506; If, execution in step 507;
Step 506 writes current line, record current line position, and the move pointer obtains the reference position of next line, returns step 505 and continues to carry out data writing operation, handle provisional capitals all in piece;
Step 507, after the data of all row in the piece all write the corresponding blocks of data of file district, file pointer is returned the piece index area reference position (being the piece reference position) on piece top, according to the line position that writes down in step 506 ablation process, in the reference position of every row write-in block index area successively; Behind the writing line index information, return step 502 and continue to handle other data blocks;
Step 508, after handling all data blocks, the index area reference position with file pointer backspace file top writes the index area successively with every reference position.The reference position of described data block is the reference position of piece index area.
In sum, the write data mode that present embodiment adopts is: calculate writing position according to line number in piece number and the piece earlier, write current line then, write down the position of every row simultaneously, and last again the line index information write-in block index area that writes down in the write data process; Equally, finish the writing of all data blocks after, just the piece index information is write the index area.This mode is a preferred process of the present invention, because each row of data may occupation space vary in size, so write index information according to actual taking up room again after earlier data being write file, can distribute suitable storage space according to the size that writes data like this.Certainly, if the size of each row of data is fixed or is changed little, also can deposit index information and report data by pre-configured fixing storage space, calculate writing position according to piece number and line number then, piece index information, line index information and each row of data are write file successively.
Preferably, when step 507 writes every start of line position in the piece index area, the line number of this piece is also together write as index information (not marking among Fig. 2 and the figure shown in Figure 3).Because behind the deblocking, the line number in last piece is less than the line number in other pieces sometimes, the line number information by preserving in the piece index area during inquiry just can know how many line data this piece has.Certainly, also the piece number can be write the index area of file if desired.
Illustrate, example in the continuation, report data has 100 row, is divided into 4, and every line number is (32,32,32,4):
1, has 4, need space 4 * 4 (preserve with 4 bytes the position of each integer representation) in order to preserve every reference position, preceding 16 bytes of file are used for preserving the piece index information like this, the data field reference position is 16 (byte representation is since 0 in the file) just, and file pointer is moved on to 16;
2, obtain first fast (preceding 1-32 is capable, and the report line numerical table shows since 1), reference position 16;
3, the reference position of 32 row needs 32 * 4 byte spaces to preserve, and increases a space 4 of preserving line number again, and the blocks of data district reference position of this part is like this: 16+128+4=148;
4, since 148, write this 32 line data successively in file, read the file pointer position before writing at every turn;
5, the move pointer is to the starting position 16 of piece, line number 32 in the write-in block in the piece index area, and every row reference position (obtaining from step 4) hereof;
6, obtain next blocks of data, and repeating step 3-5, to the last one processed;
7, the move pointer writes every reference position (obtaining) to the file reference position from step 2 in the index area; If write " block message 4 ", step 1 just needs to reserve (1+4) * 4 totally 20 byte spaces.
(3) read data
Read data is the report data that reads nominated bank or designating unit lattice from file, and according to the process of above-mentioned write data, the step of read data can be the process flow diagram that reads form in the embodiment of the invention from file with reference to Fig. 6.
Step 603 moves on to file pointer the reference position of asking row place piece;
Step 604 according to the sequence number of request row in piece, continues the piece index area of this piece of inquiry, obtains the reference position of this request row;
Step 605 moves on to the reference position of request row with file pointer, and reads this line data;
Step 606 obtains the cell of request from row.
Illustrate, suppose that request obtains certain cell data of the 54th row (the report line numerical table shows since 1):
1, the piecemeal result is: piece is several 4, every line number (32,32,32,4);
2, by piece number in the step 1 and every line number, can obtain the 54th the row second the 22nd the row;
3, the reference position of second blocks of data is kept at byte place, (2-1) * 4 in the file index district, i.e. the 4th to 7 byte; From the file index district, read this part information, obtain second blocks of data position hereof;
4, the move pointer is to the piece reference position;
5, because the row that needs is 22, so, move to the byte place of depositing the 22nd start of line position in the piece index area from piece reference position move pointer (1+ (22-1)) * 4=88 byte;
6, read the 22nd start of line positional information from this position, and the reference position of move pointer the 22nd row in the blocks of data district;
7, read the 22nd line data from this position;
8, from the row that obtains, read required cell again.
Preferably, in actual applications because the visit of corresponding report data can once visit usually with each grid in the delegation, thus present embodiment after reading data line, in internal memory buffer memory the information of delegation.When obtaining visiting the request of a grid, can check whether be certain grid of cache lines,, and do not need from file, to read earlier like this if then can directly use cache lines.
Above-mentioned caching report technology is only preserved report name (distinguishing different files by file name in hard disk) in internal memory, and report data is left in the exterior storage mediums such as hard disk, loads corresponding data according to report name from hard disk when reading at every turn.The present invention is for the visit of form, owing at every turn from the hard disk reading of data, reduced taking of internal memory, under the situation of how concurrent, big data quantity, the use that can save memory headroom in a large number improves the concurrent processing ability of system; Simultaneously, read report file from hard disk, greatly improved the efficient that file reads by this indexed mode.Therefore, the method for the invention has obtained a balance between memory headroom and access speed.Wherein, described exterior storage medium is meant the large-capacity and high-performance storage medium outside the internal memory, comprises hard disk, flash memory etc. and the following similar storage medium that may occur.
By test, on the PC of a 2GHz processor, 1G internal memory, form for one 5000 row * 15 row, if each Html page shows 30 row, then reporting system can be supported 50 concurrent requests of creating form, 50 form page turnings, create form response in average 1.5 minutes, page turning response in average 0.4 minute, and system CPU, internal memory use all only below 70%.
At above-mentioned report data caching method, the present invention also provides a kind of embodiment of report data caching system.With reference to Fig. 7, be the structured flowchart of the described caching report of embodiment system, described system mainly comprises storage unit 701, deblocking unit 702, write operation unit 703 and read operation unit 704.
Storage unit 701 is used to store report data, and what take is hard drive space.Storage unit 701 is divided into index area and data field with storage space, wherein data area stores is divided into the data of piece, the reference position that index area storage is every with reference to shown in Figure 2, can find any blocks of data in the data field according to every reference position in the index area.
Preferably, described data field is divided into piece index area and blocks of data district at every blocks of data again, wherein the blocks of data district stores all line data of this piece, the reference position of each row of data in this piece is stored in the piece index area, with reference to shown in Figure 3, can in data block, find any data line according to the data line reference position in the piece index area.
This two-layer index structure, can realize reading of report data fast, when searching certain delegation report data, just can locate fast by described two layer indexs, promptly find the piece at this line data place earlier in the file index district, find the reference position of this line data then in the piece index area of this piece, and then find this row reading of data.
Preferably, the line number information of this piece has also been preserved in described index area, uses when being used for search index.Be less than the situation of other pieces at the line number in last piece, the line number information by preserving in the piece index area during inquiry just can know how many line data this piece has.Certainly, also can in the index area, preserve the piece number if desired.
Write operation unit 703 is used for writing report data to storage unit 701, obtains Fig. 2 or storage effect shown in Figure 3.Write operation unit 703 calculates writing position according to line number in piece number and the piece earlier, writes current line then, writes down the position of every row simultaneously, and is last again the line index information write-in block index area that writes down in the write data process; Equally, finish the writing of all data blocks after, just the piece index information is write the index area.Detailed ablation process can be not described in detail in this with reference to shown in Figure 5.
Read operation unit 704 is used for reading from storage unit 701 report data of appointment, at first obtain asking row place piece and the sequence number in piece according to piece number and every line number, obtain the reference position of request msg then according to piece index and line index information, just can read the data that need from this position.The detailed process that reads can be not described in detail in this with reference to shown in Figure 6.
Preferably, also be provided with buffer unit 705, what take is memory headroom, is used for the data line that cache read operating unit 704 reads.Owing to, can once visit with each grid in the delegation usually, so in the current data that read of memory cache to the visit of report data; when certain cell in this row is read in request; directly from buffer unit, read, and do not need to read hard disk, thereby improved reading efficiency.
Preferably, the piecemeal result all can be used in write operation unit 703 and read operation unit 704, because the execution result of block algorithm also needs to take Installed System Memory, and block algorithm itself is than more efficient, and execution speed is very fast.So for save memory headroom as far as possible, write operation unit 703 and read operation unit 704 all called deblocking unit 702 and carry out a block algorithm, and can guarantee that twice piecemeal comes to the same thing before carrying out.
Above-mentioned caching report system is stored in report data in the hard disk, only deposits file name at internal memory, when needs are visited certain line data of certain form, navigates to document location in the hard disk according to file name, finds the data of needs then according to two layer indexs.Owing at every turn from the hard disk reading of data, reduced taking of internal memory, improved the concurrent processing ability of system; Simultaneously, described index stores structure has greatly improved the efficient that fixed disk file reads.Therefore, described system can reduce taking of memory headroom in the fast access report data.
The part that does not describe in detail in the system shown in Figure 7 can be considered for length referring to the relevant portion of Fig. 2-method shown in Figure 6, is not described in detail in this.And illustrated throughout is the explanation of carrying out with the behavior example, at the data with row-column configuration, the explanation that is listed as is applicable to the present invention too.
More than to a kind of data cache method provided by the present invention and system, be described in detail, used specific case herein principle of the present invention and embodiment are set forth, the explanation of above embodiment just is used for helping to understand method of the present invention and core concept thereof; Simultaneously, for one of ordinary skill in the art, according to thought of the present invention, part in specific embodiments and applications all can change.In sum, this description should not be construed as limitation of the present invention.
Claims (15)
1. a data cache method is characterized in that, comprising:
External memory space is divided into index area and data field;
With deblocking;
According to the deblocking result, every blocks of data is write the data field of described external memory space successively, the reference position of every blocks of data is write successively the index area of described external memory space;
According to the reference position of deblocking result and every blocks of data, reading of data.
2. method according to claim 1 is characterized in that: described data have row-column configuration, carry out piecemeal according to row or row during deblocking.
3. a data cache method is characterized in that, comprising:
External memory space is divided into index area and data field, and described data field is divided into piece index area and blocks of data district to data block again;
With deblocking;
According to the deblocking result, every blocks of data is write the data field of described external memory space successively, the reference position of every blocks of data is write successively the index area of described external memory space; When writing every blocks of data, each row of data is write the blocks of data district of this data block successively, the reference position of every row is write successively the piece index area of this data block;
According to the reference position of deblocking result, every blocks of data and the reference position of each row of data, reading of data.
4. method according to claim 3 is characterized in that, carries out deblocking according to following steps:
Preset largest block number and every minimum line number;
If all line numbers of data are greater than the product of largest block number and every minimum line number, then every line number equals every minimum line number and multiply by 2;
Otherwise every line number equals every minimum line number, and the data block number equals
(all line numbers+every line number-1) every line number of ÷.
5. method according to claim 3 is characterized in that the result writes data according to following steps according to piecemeal:
Count computation index district size according to piece, obtain the data field reference position;
According to line number computing block index area size in the piece, obtain blocks of data district reference position;
Write current line, write down the current line position, and obtain the reference position of next line;
After writing a blocks of data, the reference position of every row in this piece is write the piece index area of this piece;
After writing all data blocks, every reference position is write the index area of storage space.
6. method according to claim 5 is characterized in that, also comprises: the line number that every blocks of data is comprised is the write-in block index area also.
7. method according to claim 3 is characterized in that, according to the piecemeal result according to the following steps reading of data:
According to piece number and every line number, obtain asking row place piece and the sequence number in piece;
According to request row place piece, inquire about the index area of storage space, obtain the reference position of this piece;
According to the sequence number of request row in piece, continue the piece index area of this piece of inquiry, obtain the reference position of this request row;
Read line data from the reference position of this request row.
8. method according to claim 7 is characterized in that, also comprises: the data line that buffer memory reads, when cell in this row is read in request, directly read from buffer memory.
9. according to claim 4,5,6,7 described methods, it is characterized in that: described data have row-column configuration, and described according to going the data of carrying out piecemeal, by the ranks coordinate transform, replaceable one-tenth carries out the data of piecemeal according to row.
10. method according to claim 3 is characterized in that: the each execution when reading or writing data, and all carry out a deblocking operation and obtain the piecemeal result.
11. a data buffering system is characterized in that, comprising:
Storage unit is used for external memory space is divided into index area and data field, and wherein data area stores is divided into the data of piece, and the reference position of every blocks of data is stored in the index area;
The deblocking unit is used for deblocking;
Write operation unit is used for according to the deblocking result, and every blocks of data is write the data field of described external memory space successively, the reference position of every blocks of data is write successively the index area of described external memory space;
The read operation unit is used for the reference position according to deblocking result and every blocks of data, reading of data.
12. system according to claim 11 is characterized in that: described data have row-column configuration, and described deblocking unit carries out piecemeal according to row or row.
13. system according to claim 12, it is characterized in that: in the described storage unit, the data field is divided into piece index area and blocks of data district to data block again, and wherein the blocks of data district of every blocks of data stores all line data of this piece, and the reference position of every row in this piece is stored in the piece index area.
14. system according to claim 13 is characterized in that: the line number that this blocks of data comprises is also stored in the piece index area of described every blocks of data.
15. system according to claim 12 is characterized in that, also comprises: buffer unit, be used for the data line that buffer memory reads, when cell in this row is read in request, directly from buffer unit, read.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CNB200710195981XA CN100541454C (en) | 2007-12-14 | 2007-12-14 | A kind of data cache method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CNB200710195981XA CN100541454C (en) | 2007-12-14 | 2007-12-14 | A kind of data cache method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN101178693A true CN101178693A (en) | 2008-05-14 |
| CN100541454C CN100541454C (en) | 2009-09-16 |
Family
ID=39404951
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CNB200710195981XA Expired - Fee Related CN100541454C (en) | 2007-12-14 | 2007-12-14 | A kind of data cache method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN100541454C (en) |
Cited By (36)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102323908A (en) * | 2011-08-03 | 2012-01-18 | 浪潮(北京)电子信息产业有限公司 | Method and system for data caching in large volume data synchronization process on disk |
| CN102385606A (en) * | 2010-09-03 | 2012-03-21 | 腾讯科技（深圳）有限公司 | Method and device for accessing distributed data warehouse |
| CN102456024A (en) * | 2010-10-26 | 2012-05-16 | 深圳市金蝶友商电子商务服务有限公司 | Method and device for asynchronously processing newly added large data volume |
| CN102467523A (en) * | 2010-11-03 | 2012-05-23 | 英业达股份有限公司 | Method for establishing index files and method for querying data blocks by using index files |
| CN102542041A (en) * | 2011-12-27 | 2012-07-04 | 北京地拓科技发展有限公司 | Method and system for processing raster data |
| CN102694828A (en) * | 2011-03-23 | 2012-09-26 | 中兴通讯股份有限公司 | Method and apparatus for data access in distributed caching system |
| CN102722484A (en) * | 2011-03-29 | 2012-10-10 | 新奥特（北京）视频技术有限公司 | A file-buffering method, a device thereof and application thereof |
| CN102739693A (en) * | 2011-03-31 | 2012-10-17 | 新奥特（北京）视频技术有限公司 | Method and apparatus for distributing statements used in sports event information system |
| CN102833769A (en) * | 2012-09-20 | 2012-12-19 | 苏州坤里达信息科技有限公司 | Efficient management method for interference matrix in wireless communication network system |
| CN102880615A (en) * | 2011-07-15 | 2013-01-16 | 腾讯科技（深圳）有限公司 | Data storage method and device |
| CN102968456A (en) * | 2012-10-30 | 2013-03-13 | 北京地拓科技发展有限公司 | Method and device for reading and processing raster data |
| CN103269383A (en) * | 2013-06-14 | 2013-08-28 | 苏州海葵电子科技有限公司 | Method for realizing communication of train network |
| CN103500224A (en) * | 2013-10-18 | 2014-01-08 | 税友软件集团股份有限公司 | Data writing method and device and data reading method and device |
| WO2014015828A1 (en) * | 2012-07-27 | 2014-01-30 | 腾讯科技（深圳）有限公司 | Data storage space processing method and processing system, and data storage server |
| CN103560976A (en) * | 2013-11-20 | 2014-02-05 | 迈普通信技术股份有限公司 | Method, device and system for having control over data sending |
| CN103631726A (en) * | 2013-11-28 | 2014-03-12 | 北京锐安科技有限公司 | File processing method and device of series-connection streaming computational nodes |
| CN103942289A (en) * | 2014-04-12 | 2014-07-23 | 广西师范大学 | Memory caching method oriented to range querying on Hadoop |
| CN104484332A (en) * | 2014-11-11 | 2015-04-01 | 珠海天琴信息科技有限公司 | Method and device for reading and writing data in embedded system |
| CN104699815A (en) * | 2015-03-24 | 2015-06-10 | 北京嘀嘀无限科技发展有限公司 | Data processing method and system |
| CN103092775B (en) * | 2013-01-31 | 2015-06-10 | 武汉大学 | Spatial data double cache method and mechanism based on key value structure |
| CN105912274A (en) * | 2016-04-21 | 2016-08-31 | 乐视控股（北京）有限公司 | Streaming data positioning method and apparatus |
| CN106326499A (en) * | 2016-10-14 | 2017-01-11 | 广州市千钧网络科技有限公司 | Data processing method and device |
| CN107197116A (en) * | 2017-05-25 | 2017-09-22 | 天津大学 | One kind is based on the real-time reliable graph of udp protocol as transmission plan |
| CN107506308A (en) * | 2017-06-30 | 2017-12-22 | 杭州旗捷科技有限公司 | A kind of chip rewrites date storage method, electronic equipment and the storage medium of equipment |
| CN107562705A (en) * | 2017-07-25 | 2018-01-09 | 努比亚技术有限公司 | A kind of data export method, service end and computer-readable recording medium |
| WO2018077219A1 (en) * | 2016-10-27 | 2018-05-03 | 深圳创维数字技术有限公司 | Memory management method and system |
| CN108153874A (en) * | 2017-12-26 | 2018-06-12 | 福建星瑞格软件有限公司 | A kind of big data height takes the quick paging method of query results |
| WO2019072088A1 (en) * | 2017-10-13 | 2019-04-18 | 杭州海康威视系统技术有限公司 | File management method, file management device, electronic equipment and storage medium |
| CN109669622A (en) * | 2017-10-13 | 2019-04-23 | 杭州海康威视系统技术有限公司 | A kind of file management method, document management apparatus, electronic equipment and storage medium |
| CN110311954A (en) * | 2019-05-31 | 2019-10-08 | 上海赫千电子科技有限公司 | The data reading system and its read method of automobile sensor |
| CN110874283A (en) * | 2018-08-31 | 2020-03-10 | 杭州海康威视系统技术有限公司 | Data storage method and device based on erasure codes and electronic equipment |
| CN111143284A (en) * | 2018-11-02 | 2020-05-12 | 浙江宇视科技有限公司 | Dynamic indexing method and device for file system |
| CN111400427A (en) * | 2019-01-02 | 2020-07-10 | 阿里巴巴集团控股有限公司 | Data storage method, data query method, data storage device, data query device and computing equipment |
| CN111552899A (en) * | 2020-03-20 | 2020-08-18 | 广联达科技股份有限公司 | Method and system for improving display performance of front-end report |
| WO2021068564A1 (en) * | 2019-10-12 | 2021-04-15 | 平安科技（深圳）有限公司 | Data processing method and apparatus, and device and storage medium |
| CN112860595A (en) * | 2021-03-12 | 2021-05-28 | 湖南国科微电子股份有限公司 | PCI (peripheral component interconnect express) equipment or PCIE (peripheral component interconnect express) equipment, data access method and related assembly |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102722555A (en) * | 2012-05-28 | 2012-10-10 | 北京网尚数字电影院线有限公司 | Method and system for caching multimedia file |
-
2007
- 2007-12-14 CN CNB200710195981XA patent/CN100541454C/en not_active Expired - Fee Related
Cited By (55)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102385606A (en) * | 2010-09-03 | 2012-03-21 | 腾讯科技（深圳）有限公司 | Method and device for accessing distributed data warehouse |
| CN102385606B (en) * | 2010-09-03 | 2016-08-03 | 腾讯科技（深圳）有限公司 | The access method of a kind of Distributed Data Warehouse and device |
| CN102456024A (en) * | 2010-10-26 | 2012-05-16 | 深圳市金蝶友商电子商务服务有限公司 | Method and device for asynchronously processing newly added large data volume |
| CN102467523A (en) * | 2010-11-03 | 2012-05-23 | 英业达股份有限公司 | Method for establishing index files and method for querying data blocks by using index files |
| CN102694828A (en) * | 2011-03-23 | 2012-09-26 | 中兴通讯股份有限公司 | Method and apparatus for data access in distributed caching system |
| CN102694828B (en) * | 2011-03-23 | 2016-03-30 | 中兴通讯股份有限公司 | A kind of method of distributed cache system data access and device |
| CN102722484A (en) * | 2011-03-29 | 2012-10-10 | 新奥特（北京）视频技术有限公司 | A file-buffering method, a device thereof and application thereof |
| CN102722484B (en) * | 2011-03-29 | 2017-12-29 | 新奥特（北京）视频技术有限公司 | A kind of file buffering method, apparatus and its application |
| CN102739693A (en) * | 2011-03-31 | 2012-10-17 | 新奥特（北京）视频技术有限公司 | Method and apparatus for distributing statements used in sports event information system |
| CN102880615A (en) * | 2011-07-15 | 2013-01-16 | 腾讯科技（深圳）有限公司 | Data storage method and device |
| CN102323908A (en) * | 2011-08-03 | 2012-01-18 | 浪潮(北京)电子信息产业有限公司 | Method and system for data caching in large volume data synchronization process on disk |
| CN102542041A (en) * | 2011-12-27 | 2012-07-04 | 北京地拓科技发展有限公司 | Method and system for processing raster data |
| CN103577339A (en) * | 2012-07-27 | 2014-02-12 | 深圳市腾讯计算机系统有限公司 | Method and system for storing data |
| WO2014015828A1 (en) * | 2012-07-27 | 2014-01-30 | 腾讯科技（深圳）有限公司 | Data storage space processing method and processing system, and data storage server |
| CN103577339B (en) * | 2012-07-27 | 2018-01-30 | 深圳市腾讯计算机系统有限公司 | A kind of date storage method and system |
| US9323685B2 (en) | 2012-07-27 | 2016-04-26 | Tencent Technology (Shenzhen) Company Limited | Data storage space processing method and processing system, and data storage server |
| CN102833769A (en) * | 2012-09-20 | 2012-12-19 | 苏州坤里达信息科技有限公司 | Efficient management method for interference matrix in wireless communication network system |
| CN102968456B (en) * | 2012-10-30 | 2016-08-24 | 北京地拓科技发展有限公司 | A kind of raster data reading and processing method and device |
| CN102968456A (en) * | 2012-10-30 | 2013-03-13 | 北京地拓科技发展有限公司 | Method and device for reading and processing raster data |
| CN103092775B (en) * | 2013-01-31 | 2015-06-10 | 武汉大学 | Spatial data double cache method and mechanism based on key value structure |
| CN103269383A (en) * | 2013-06-14 | 2013-08-28 | 苏州海葵电子科技有限公司 | Method for realizing communication of train network |
| CN103500224B (en) * | 2013-10-18 | 2016-03-16 | 税友软件集团股份有限公司 | A kind of method for writing data and device, method for reading data and device |
| CN103500224A (en) * | 2013-10-18 | 2014-01-08 | 税友软件集团股份有限公司 | Data writing method and device and data reading method and device |
| CN103560976A (en) * | 2013-11-20 | 2014-02-05 | 迈普通信技术股份有限公司 | Method, device and system for having control over data sending |
| CN103560976B (en) * | 2013-11-20 | 2018-12-07 | 迈普通信技术股份有限公司 | A kind of method, apparatus and system that control data are sent |
| CN103631726A (en) * | 2013-11-28 | 2014-03-12 | 北京锐安科技有限公司 | File processing method and device of series-connection streaming computational nodes |
| CN103631726B (en) * | 2013-11-28 | 2017-01-25 | 北京锐安科技有限公司 | File processing method and device of series-connection streaming computational nodes |
| CN103942289A (en) * | 2014-04-12 | 2014-07-23 | 广西师范大学 | Memory caching method oriented to range querying on Hadoop |
| CN103942289B (en) * | 2014-04-12 | 2017-01-25 | 广西师范大学 | Memory caching method oriented to range querying on Hadoop |
| CN104484332A (en) * | 2014-11-11 | 2015-04-01 | 珠海天琴信息科技有限公司 | Method and device for reading and writing data in embedded system |
| CN104699815A (en) * | 2015-03-24 | 2015-06-10 | 北京嘀嘀无限科技发展有限公司 | Data processing method and system |
| CN105912274A (en) * | 2016-04-21 | 2016-08-31 | 乐视控股（北京）有限公司 | Streaming data positioning method and apparatus |
| WO2017181614A1 (en) * | 2016-04-21 | 2017-10-26 | 乐视控股（北京）有限公司 | Streaming data positioning method, apparatus and electronic device |
| CN106326499B (en) * | 2016-10-14 | 2019-10-18 | 广州市千钧网络科技有限公司 | A kind of data processing method and device |
| CN106326499A (en) * | 2016-10-14 | 2017-01-11 | 广州市千钧网络科技有限公司 | Data processing method and device |
| WO2018077219A1 (en) * | 2016-10-27 | 2018-05-03 | 深圳创维数字技术有限公司 | Memory management method and system |
| CN107197116A (en) * | 2017-05-25 | 2017-09-22 | 天津大学 | One kind is based on the real-time reliable graph of udp protocol as transmission plan |
| CN107506308A (en) * | 2017-06-30 | 2017-12-22 | 杭州旗捷科技有限公司 | A kind of chip rewrites date storage method, electronic equipment and the storage medium of equipment |
| CN107562705A (en) * | 2017-07-25 | 2018-01-09 | 努比亚技术有限公司 | A kind of data export method, service end and computer-readable recording medium |
| CN109669623A (en) * | 2017-10-13 | 2019-04-23 | 杭州海康威视系统技术有限公司 | A kind of file management method, document management apparatus, electronic equipment and storage medium |
| CN109669622A (en) * | 2017-10-13 | 2019-04-23 | 杭州海康威视系统技术有限公司 | A kind of file management method, document management apparatus, electronic equipment and storage medium |
| WO2019072088A1 (en) * | 2017-10-13 | 2019-04-18 | 杭州海康威视系统技术有限公司 | File management method, file management device, electronic equipment and storage medium |
| CN109669623B (en) * | 2017-10-13 | 2021-09-03 | 杭州海康威视系统技术有限公司 | File management method, file management device, electronic equipment and storage medium |
| CN108153874A (en) * | 2017-12-26 | 2018-06-12 | 福建星瑞格软件有限公司 | A kind of big data height takes the quick paging method of query results |
| CN110874283B (en) * | 2018-08-31 | 2021-07-20 | 杭州海康威视系统技术有限公司 | Data storage method and device based on erasure codes and electronic equipment |
| CN110874283A (en) * | 2018-08-31 | 2020-03-10 | 杭州海康威视系统技术有限公司 | Data storage method and device based on erasure codes and electronic equipment |
| CN111143284A (en) * | 2018-11-02 | 2020-05-12 | 浙江宇视科技有限公司 | Dynamic indexing method and device for file system |
| CN111143284B (en) * | 2018-11-02 | 2023-06-20 | 浙江宇视科技有限公司 | Dynamic indexing method and device for file system |
| CN111400427A (en) * | 2019-01-02 | 2020-07-10 | 阿里巴巴集团控股有限公司 | Data storage method, data query method, data storage device, data query device and computing equipment |
| CN110311954A (en) * | 2019-05-31 | 2019-10-08 | 上海赫千电子科技有限公司 | The data reading system and its read method of automobile sensor |
| CN110311954B (en) * | 2019-05-31 | 2022-02-15 | 上海赫千电子科技有限公司 | Data reading system and method for vehicle sensor |
| WO2021068564A1 (en) * | 2019-10-12 | 2021-04-15 | 平安科技（深圳）有限公司 | Data processing method and apparatus, and device and storage medium |
| CN111552899A (en) * | 2020-03-20 | 2020-08-18 | 广联达科技股份有限公司 | Method and system for improving display performance of front-end report |
| CN111552899B (en) * | 2020-03-20 | 2023-06-23 | 广联达科技股份有限公司 | Method and system for improving display performance of front-end report |
| CN112860595A (en) * | 2021-03-12 | 2021-05-28 | 湖南国科微电子股份有限公司 | PCI (peripheral component interconnect express) equipment or PCIE (peripheral component interconnect express) equipment, data access method and related assembly |
Also Published As
| Publication number | Publication date |
|---|---|
| CN100541454C (en) | 2009-09-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN100541454C (en) | A kind of data cache method and system | |
| JP6815378B2 (en) | Distributed storage system | |
| US20180285167A1 (en) | Database management system providing local balancing within individual cluster node | |
| CN107423422B (en) | Spatial data distributed storage and search method and system based on grid | |
| CN100383792C (en) | Buffer data base data organization method | |
| CN110058822B (en) | Transverse expansion method for disk array | |
| CN106775476A (en) | Mixing memory system and its management method | |
| CN103530387A (en) | Improved method aimed at small files of HDFS | |
| CN102411616B (en) | Method and system for storing data and data management method | |
| US20130198453A1 (en) | Hybrid storage device inclucing non-volatile memory cache having ring structure | |
| WO2009114218A2 (en) | Web-based multiuser collaboration | |
| CN102591947A (en) | Fast and low-RAM-footprint indexing for data deduplication | |
| CN103597482A (en) | Storing data on storage nodes | |
| JP2006268776A (en) | Memory card | |
| CN104115133A (en) | Data migration for composite non-volatile storage device | |
| CN101373445B (en) | Method and apparatus for scheduling memory | |
| CN102521419A (en) | Hierarchical storage realization method and system | |
| CN103116661A (en) | Data processing method of database | |
| CN105138282A (en) | Storage space recycling method and storage system | |
| CN108733306A (en) | A kind of Piece file mergence method and device | |
| CN110297601B (en) | Solid state disk array construction method, electronic device and storage medium | |
| CN103019884A (en) | Memory page de-weight method and memory page de-weight device based on virtual machine snapshot | |
| CN110795363B (en) | Hot page prediction method and page scheduling method of storage medium | |
| CN104111898A (en) | Hybrid storage system based on multidimensional data similarity and data management method | |
| CN111708894B (en) | Knowledge graph creation method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20090916 |