CN101221577A - Method for saving and loading internal memory data base tables - Google Patents
Method for saving and loading internal memory data base tables Download PDFInfo
- Publication number
- CN101221577A CN101221577A CNA2008100042460A CN200810004246A CN101221577A CN 101221577 A CN101221577 A CN 101221577A CN A2008100042460 A CNA2008100042460 A CN A2008100042460A CN 200810004246 A CN200810004246 A CN 200810004246A CN 101221577 A CN101221577 A CN 101221577A
- Authority
- CN
- China
- Prior art keywords
- data
- compression
- memory
- compression blocks
- file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method for realizing saving to disk of memory database tables which comprises that: 1) when a memory database table is saved to disk, the memory table data with a record is loaded to a memory buffer area; 2) a table saving file is created and table registration information is written in; 3) the checksum of the data of the memory table is calculated; 4) the data of the memory table is made a blocking compression, and the compression result and the checksum are written into the corresponding table saving file. The invention also discloses a memory database table loading method for realizing the saving of the method, which comprises that: a) the data of a registration structure of the memory table is read; b) the compressed memory table data of an appointed table is read to the memory buffer area in a blocking mode, and is decompressed; c) after the entire blocking data is decompressed, the data in the memory buffer area is loaded into the memory table. The proposal of the invention can save according to the content of the actual table data of the memory table, makes a blocking compression processing during the saving, saves the space of the magnetic disk, and quickens the speed of the saving.
Description
Technical field
The present invention relates to the database of moving communicating field, relate in particular to a kind of main memory database table deposit and method that loads of realizing.
Background technology
In mobile communication technology is used, need to create a plurality of memory tables to deposit the needed data configuration information of service application.In order to preserve configuration data, configuration data need be deposited.When other veneer or system need, can load and do not need to reconfigure again from disk.Along with the expansion of service application function, the required data volume of database application increases severely, and embodies both ways: the one, and the table number is more and more, and the 2nd, the table capacity extensions, internal memory, the external space of consumption are increasing.
The deposit mechanism of former database adopts non-compression mechanism to file, and the history file size is the greatest requirements space sum of each tables of data, and is irrelevant with the physical record number of concrete table.Individual tables of data even should show the no datat record in the internal memory, after the deposit, in the database history file, also takies the peripheral hardware space by maximum table capacity.The data extending that causes along with present application, and under the situation that disk size does not increase, can be fewer and feweri on disk for the remaining space of database deposit.Peripheral hardware capacity cause inadequately the depositing fault of failure repeatedly appears in use in the outfield.
Summary of the invention
The invention provides a kind of main memory database table deposit and method that loads of realizing, to solve the problem of disk size deficiency.
In order to solve the problems of the technologies described above, the invention provides a kind of method that realizes the main memory database table deposit, comprise the steps:
(1) when carrying out the main memory database table deposit, the memory table data load that record will be arranged is to core buffer;
(2) establishment table saving file and write the table log-on message in the saving file to table;
(3) calculate described memory table data verification and;
(4) described memory table data are carried out piecemeal compression and with compression result and described verification with write the table saving file of correspondence.
The method of the invention, wherein,
In the step (4), when writing data with the piecemeal compress mode, calculate the size of compression blocks according to the principle of dividing the integrality of holding the record when determining: according to the record length of maximum compression block size divided by table, obtain maximum record numbers, write down number with this then and multiply by the size that record length is obtained compression blocks; Described compression result comprises compression blocks size and compression blocks number.
The method of the invention, wherein,
In the step (2), be the saving file name with the table name during establishment table saving file, and add a file suffixes.
The method of the invention, wherein,
In the step (4), write block by block by side-play amount in the deposit process, all piecemeal is delayed time after writing.
Further,
Structure after the described main memory database table deposit comprises:
Data N, the number of expression compression blocks;
Data check and;
The file availability structure is used to write down the size of each N compression blocks;
The data of each compression blocks, wherein first compression blocks is the log-on message compression blocks of the structure explanation of table, is thereafter each compression blocks of the data of table.
Further, described N is 0 or surpasses maximum compression piece number all to be considered as these data invalid.
The present invention also provides a kind of method that realizes the main memory database table loading of method deposit as mentioned above, comprises the steps:
(a) read memory table registration structured data;
(b) piecemeal read named list compression memory table data to core buffer, and decompress;
(c) treat that all block datas all decompress and finish, with the data load of core buffer in memory table.
The method of loading of the present invention, wherein,
In the step (a), according to the length of compression blocks number calculation document validity structured data, length reads the file availability structured data according to this then, promptly data check and with each compression blocks size.
The method of loading of the present invention, wherein,
In the step (c), in loading procedure, with the contrast of the maximum amount of data in the log-on message of the log-on message shown in the buffer zone and current table, if the long-pending table capacity and the record length greater than current table of record number of showing in the buffer zone and record length is long-pending, then the data volume with current table is as the criterion as needs loaded data amount, otherwise still with the data volume in the disk as needs loaded data amount.
The method of loading of the present invention, wherein,
In the step (c), the table data behind the decompress(ion) are carried out verification and calculating,,, think that data are correct if consistent with result of calculation and the verification and the comparison of reading, otherwise error in data.
Scheme of the present invention can be deposited according to the actual table data content of memory table, and in addition the piecemeal compression is handled in the deposit process, has saved disk space, has accelerated the speed of deposit, also can carry out the processing of version rollback.
Description of drawings
Fig. 1 is the deposit structural representation of a kind of disk file of application example of the present invention;
Fig. 2 is the schematic flow sheet that application example of the present invention is written to the data content of a table disk;
Fig. 3 is the schematic flow sheet that application example of the present invention is loaded into the disk file of depositing among Fig. 2 core buffer.
Embodiment
The present invention proposes the method for carrying out submeter deposit by the real data of table: when deposit, only the table that record is arranged is deposited, and by the real data processing of depositing.The his-and-hers watches data content compresses processing in the deposit process.The too high problem of occupancy of CPU when compressing with the decompression big data quantity in order to solve, determine that the data volume of table is carried out piecemeal according to certain size to be handled, whenever handle the laggard line delay of a block data, to reduce the occupancy of CPU, the data block-by-block after will compressing then writes disk.The effect of Chu Liing is to have saved disk space like this, has increased the deposit speed of table, has reduced the occupancy of CPU.
Below in conjunction with the drawings and specific embodiments technical scheme of the present invention is described in detail.
The enforcement of this programme comprises data deposit and data load.Data deposits is with the processing of depositing of memory table data structure and recorded content, and data load is that the saving file on the disk is loaded in the memory table in the memory database.
As shown in Figure 1, be the deposit structural representation of a kind of disk file of application example of the present invention, at first be the data N of 2 bytes, N is 0 or surpasses maximum compression piece number all to be considered as these data invalid, otherwise the number of expression compression blocks; Be then 4 bytes data check and, next is used to write down the size of each N compression blocks, secondly is the data of each compression blocks again.Wherein first compression blocks is the log-on message compression blocks of the structure explanation of table, is thereafter each compression blocks of the data of table.
As shown in Figure 2, application example of the present invention is written to the schematic flow sheet of disk with the data content of a table, comprises the steps:
Step 201: when carrying out the main memory database table deposit, memory table is loaded into core buffer;
Step 202: the deposit catalogue and the filename that obtain the table place;
Step 203: judge whether current table has record; As do not have, then change step 204; If any, change step 205;
Step 204: current file in the deletion disk, flow process finishes;
Step 205: the compression blocks number when calculating the deposit of current table: the principle of the integrality of holding the record during according to piecemeal is calculated the size of compression blocks: obtain the record number with the maximum compression piece divided by the length of table record, multiply by the size that record length obtains compression blocks with the record number again; Be standard by this size then, the data volume of the table of being deposited by needs is calculated possible compression blocks number;
Step 206: create the table saving file; Be the saving file name with the table name during establishment table saving file, and add a file suffixes;
Step 208: write 0 at file header, the expression begin the deposit, and calculate this table data verification and;
Step 209: the log-on message that writes table with compress mode: press the data length of compression blocks number N calculation document validity, i.e. the file verification of 4 bytes and add the compression blocks size sum of N 2 bytes; The log-on message of his-and-hers watches compresses then, compression result is write in first compression blocks of disk;
Step 210: the data message that writes table with compress mode: the data message of his-and-hers watches compresses by compression block size piecemeal, read every data volume successively, and it is compressed, every compression result is write disk according to the side-play amount of every blocks of data, all successfully write disk until all data messages, size after each piece of record compresses in the file availability structure simultaneously, all piecemeals are delayed time after writing:
Step 211: with verification and, together with the compression blocks size write the table saving file in;
Step 212: the compression blocks number is write in the file header;
As shown in Figure 3, being application example of the present invention is loaded into the schematic flow sheet of core buffer with the disk file of depositing among Fig. 2, comprises the steps:
Step 301: open disk file;
Step 302: the data that read file header 2 bytes;
Step 303: judge file header data whether legal (judging that promptly whether the file header data are 0 or surpass maximum compression piece number): if these data are 0 or surpass maximum compression piece number that then these data are illegal, change step 309; If these data are legal, then change step 304;
Step 304: according to the length of compression blocks number calculation document validity structured data, length reads the file availability structured data according to this then, promptly data check and with each compression blocks size;
Step 305: read the log-on message compression blocks of table, i.e. first compression blocks data, the compression blocks of Biao registration structure just, and, the decompress(ion) result is left in the log-on message of showing in the buffer zone to its decompress(ion);
Step 306: calculate the data volume that needs read: the maximum amount of data in the log-on message of the log-on message shown in the buffer zone and current table is contrasted according to log-on message, if long-pending the table capacity and the record length long-pending (this is owing to the inconsistent reason of version causes) of record number of showing in the buffer zone and record length greater than current table, then the data volume with current table is as the criterion as needs loaded data amount, otherwise still with the data volume in the disk as needs loaded data amount;
Step 307: putting the data volume variables D ataTotal that has read is 0, the reading of data compression blocks, and to its decompress(ion), DataTotal adds the size after the decompression, if DataTotal is less than needs loaded data amount, then copy compression result to buffer zone, repeat this operation, read until all compressed data blocks and finish;
Step 308: the table data behind the decompress(ion) are carried out verification and calculating,,, think that then data are correct, load flow process and finish if consistent with result of calculation and the verification and the comparison of reading; If inconsistent, then think and change step 309 by error in data;
Step 309: close file, flow process finishes.
Scheme of the present invention is not restricted to listed utilization in instructions and the embodiment.Concerning the technology of the present invention those of ordinary skill in the field, can make various corresponding changes and distortion according to the present invention, and all these corresponding changes and distortion all belong to the protection domain of claim of the present invention.
Claims (10)
1. a method that realizes the main memory database table deposit is characterized in that, comprises the steps:
(1) when carrying out the main memory database table deposit, the memory table data load that record will be arranged is to core buffer;
(2) establishment table saving file and write the table log-on message in the saving file to table;
(3) calculate described memory table data verification and;
(4) described memory table data are carried out piecemeal compression and with compression result and described verification with write the table saving file of correspondence.
2. method according to claim 1 is characterized in that,
In the step (4), when writing data with the piecemeal compress mode, the principle of the integrality of holding the record during according to piecemeal is calculated the size of compression blocks: according to the record length of maximum compression block size divided by table, obtain maximum record numbers, write down number with this then and multiply by the size that record length is obtained compression blocks; Described compression result comprises compression blocks size and compression blocks number.
3. method according to claim 1 is characterized in that,
In the step (2), be the saving file name with the table name during establishment table saving file, and add a file suffixes.
4. method according to claim 1 is characterized in that,
In the step (4), write block by block by side-play amount in the deposit process, all piecemeal is delayed time after writing.
5. as method as described in the claim 1～4, it is characterized in that the structure after the described main memory database table deposit comprises:
Data N, the number of expression compression blocks;
Data check and;
The file availability structure is used to write down the size of each N compression blocks;
The data of each compression blocks, wherein first compression blocks is the log-on message compression blocks of the structure explanation of table, is thereafter each compression blocks of the data of table.
6. as method as described in the claim 5, it is characterized in that described N is 0 or surpasses maximum compression piece number all to be considered as these data invalid.
7. a method that realizes the main memory database table loading of the described method deposit of claim 1 is characterized in that, comprises the steps:
(a) read memory table registration structured data;
(b) piecemeal read named list compression memory table data to core buffer, and decompress;
(c) treat that all block datas all decompress and finish, with the data load of core buffer in memory table.
8. as method as described in the claim 7, it is characterized in that,
In the step (a), according to the length of compression blocks number calculation document validity structured data, length reads the file availability structured data according to this then, promptly data check and with each compression blocks size.
9. as method as described in the claim 7, it is characterized in that,
In the step (c), in loading procedure, with the contrast of the maximum amount of data in the log-on message of the log-on message shown in the buffer zone and current table, if the long-pending table capacity and the record length greater than current table of record number of showing in the buffer zone and record length is long-pending, then the data volume with current table is as the criterion as needs loaded data amount, otherwise still with the data volume in the disk as needs loaded data amount.
10. as method as described in the claim 8, it is characterized in that,
In the step (c), the table data behind the decompress(ion) are carried out verification and calculating,,, think that data are correct if consistent with result of calculation and the verification and the comparison of reading, otherwise error in data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CNB2008100042460A CN100550023C (en) | 2008-01-24 | 2008-01-24 | A kind of main memory database table deposit and method that loads of realizing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CNB2008100042460A CN100550023C (en) | 2008-01-24 | 2008-01-24 | A kind of main memory database table deposit and method that loads of realizing |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN101221577A true CN101221577A (en) | 2008-07-16 |
| CN100550023C CN100550023C (en) | 2009-10-14 |
Family
ID=39631416
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CNB2008100042460A Expired - Fee Related CN100550023C (en) | 2008-01-24 | 2008-01-24 | A kind of main memory database table deposit and method that loads of realizing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN100550023C (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103500224A (en) * | 2013-10-18 | 2014-01-08 | 税友软件集团股份有限公司 | Data writing method and device and data reading method and device |
| CN104408154A (en) * | 2014-12-04 | 2015-03-11 | 华为技术有限公司 | Repeated data deletion method and device |
| CN113312322A (en) * | 2021-04-21 | 2021-08-27 | 厦门芯泰达集成电路有限公司 | STDF file reading and writing method, medium, equipment and device |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6006232A (en) * | 1997-10-21 | 1999-12-21 | At&T Corp. | System and method for multirecord compression in a relational database |
| JP2001291048A (en) * | 2000-04-04 | 2001-10-19 | Taabo Data Laboratory Kk | Data tabulation method and storage medium stored with program for the data tabulation method |
| CN1311338C (en) * | 2004-08-23 | 2007-04-18 | 华为技术有限公司 | Method for reducing holding of internal memory in using table |
-
2008
- 2008-01-24 CN CNB2008100042460A patent/CN100550023C/en not_active Expired - Fee Related
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103500224A (en) * | 2013-10-18 | 2014-01-08 | 税友软件集团股份有限公司 | Data writing method and device and data reading method and device |
| CN103500224B (en) * | 2013-10-18 | 2016-03-16 | 税友软件集团股份有限公司 | A kind of method for writing data and device, method for reading data and device |
| CN104408154A (en) * | 2014-12-04 | 2015-03-11 | 华为技术有限公司 | Repeated data deletion method and device |
| CN104408154B (en) * | 2014-12-04 | 2018-05-29 | 华为技术有限公司 | Data de-duplication method and device |
| CN113312322A (en) * | 2021-04-21 | 2021-08-27 | 厦门芯泰达集成电路有限公司 | STDF file reading and writing method, medium, equipment and device |
| CN113312322B (en) * | 2021-04-21 | 2022-04-01 | 厦门芯泰达集成电路有限公司 | STDF file reading and writing method, medium, equipment and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN100550023C (en) | 2009-10-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8380900B2 (en) | Systems and methods for compressing files for storage and operation on compressed files | |
| US20090234870A1 (en) | Ordering compression and deduplication of data | |
| CN101916228B (en) | Flash translation layer (FTL) with data compression function and implementation method | |
| US9477682B1 (en) | Parallel compression of data chunks of a shared data object using a log-structured file system | |
| CN103136243B (en) | File system duplicate removal method based on cloud storage and device | |
| US8015220B1 (en) | Block-based differencing algorithm | |
| CN104462141B (en) | Method, system and the storage engines device of a kind of data storage and inquiry | |
| CN104123300A (en) | Data distributed storage system and method | |
| CN103020205A (en) | Compression and decompression method based on hardware accelerator card on distributive-type file system | |
| CN101261825B (en) | A word library management method for mobile terminal system | |
| CN103177077B (en) | The storage of a kind of APK file and output intent | |
| CN111061431B (en) | Distributed storage method, server and client | |
| US20120265924A1 (en) | Elastic data techniques for managing cache storage using ram and flash-based memory | |
| CN101977061A (en) | Lossless compression and decompression method for general information | |
| CN104281528A (en) | Data storage method and device | |
| CN101197773B (en) | Packet compressed transmitting and received de-compression method and system based on multilink | |
| CN100550023C (en) | A kind of main memory database table deposit and method that loads of realizing | |
| CN101707633A (en) | Message-oriented middleware persistent message storing method based on file system | |
| CN107402870A (en) | The processing method and processing device of daily record section in a kind of meta data server | |
| CN114564457B (en) | Storage space optimization method and system for database files | |
| CN112765112A (en) | Installation package packing and unpacking method | |
| CN101526959B (en) | Data storing method and device | |
| CN105677579A (en) | Data access method and system in cache system | |
| CN111309528A (en) | Data collaborative backup system and method based on cloud computing and distributed storage | |
| CN107436769A (en) | The amending method and device of a kind of cached configuration |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20190128 Address after: Delaware Patentee after: Open Invention Network Co.,Ltd. Address before: 518057 Department of law, Zhongxing building, South hi tech Industrial Park, Nanshan District hi tech Industrial Park, Guangdong, Shenzhen Patentee before: ZTE Corp. |
|
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20091014 Termination date: 20220124 |