CN1851691A - Database back-up data compression and search method - Google Patents
Database back-up data compression and search method Download PDFInfo
- Publication number
- CN1851691A CN1851691A CN 200510064796 CN200510064796A CN1851691A CN 1851691 A CN1851691 A CN 1851691A CN 200510064796 CN200510064796 CN 200510064796 CN 200510064796 A CN200510064796 A CN 200510064796A CN 1851691 A CN1851691 A CN 1851691A
- Authority
- CN
- China
- Prior art keywords
- database
- data
- file
- compression
- compress
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention provides a novel compress mode capable of raising data base structured data compressibility from current 2-3 times to 10-30 times, greatly reducing backing up data storage capacity and transmitting flow rate. The present invention also provides a simple searching method directly to save database files without operating database recovery, greatly raising data base backing up data retrieval efficiency.
Description
Technical field
Technical field of the present invention belongs to the DB Backup technology.
Background technology
The mass data of database online query generally all needs schedule backup.The data of database backup generally has two purposes: the one, for the security consideration of Database Systems, when Database Systems break down, recover with Backup Data; The 2nd, the time is inquired about the little data of frequency for a long time, file storage, so that still can retrieve when needing.
To the Backup Data of database, generally by backup storage device for example the compression chip on the magnetic tape station compress.The memory capacity of compression chip is generally less, 32KB for example, carrying out backed up data also compresses by the corresponding data block size, and the compression of data block, be regardless of data type (text, image, or packed data etc. not), adopt without exception with a kind of compressed encoding, for example LZ compressed encoding etc.
When backup storage device self does not provide the chip compression, present backup software also can provide the software compression, but the mode of compression is also similar with the chip compression, is also undertaken by less data block, and also do not distinguish data type, adopt with a kind of compressed encoding without exception.
Existing data backup system, no matter above-mentioned chip compress mode or software compress mode, all has only very low compressibility, usually doubly less than 2-3, and,, carry out the compression of LZ and so on again as jpeg picture etc. to the data of having compressed, do not have any compression effectiveness yet, can take the compression processing time by whitecap.
Except above-mentioned data compression problem, to the retrieval of Backup Data, also lack simple, fast means at present.To the retrieval of Backup Data, existing mode is at first it to be returned in the corresponding service database of online query, is deleted from the associated databases of online query after the inquiry again.The above-mentioned course of work needs database server to carry out a series of operation, and is both very loaded down with trivial details time-consuming, increases the probability of makeing mistakes again easily.
Purpose of the present invention provides the efficient compression and the real-time search method of database back-up data, exactly to improve the above-mentioned defective of present DB Backup technology greatly.
Summary of the invention
Main contents of the present invention comprise two parts:
1, to the Backup Data of database, do not adopt the small data piece compress mode in the present data backup system, and be the compression process object with the database file, and to the data of different types library file, automatically adopt different compress techniques, thereby can greatly improve the compressibility of Backup Data, significantly reduce the memory capacity and the delivery flow rate of Backup Data.
2, to the retrieval of the Backup Data of database filing storage, the database recovery operation of not wasting time and energy, the but directly database file on the Backup Data server is retrieved, thus can greatly improve the search efficiency of database back-up data.
Description of drawings
Fig. 1 is system logic structure figure of the present invention;
Fig. 2 is the database file compress mode example of a kind of simplification of the present invention;
Fig. 3 is the example of the database file catalogue of a kind of tree structure of the present invention.
Embodiment
System logic structure figure of the present invention, as shown in Figure 1.By the retrieval of data of database backup and database back-up data, be specifically described respectively below.
First: data of database backup.Carry out as follows:
1, the data of each database table that will be drawn up part by backup software export to respectively in the buffer zone with direct-connected compression service device of database server or database server self, form the corresponding database file.For example database table A forms database file A, and database table B forms database file B or the like.
Structured database table for the field formula, be convenient later retrieval process, can be when the data of carrying out above-mentioned database table derive, also copy out its structural information (field name in the lump, field length, field data types and constraint condition etc.), the structural information of database table is formed database file with data message.
2, the compressed software in compression service device or the database server to established database file, according to the difference of its file type, adopts different compress techniques to compress respectively.
The compression of database file generally can be divided into three classifications and carry out: the one, and the file that does not compress, for example file that need not compress or the file that carries out overcompression; The one, should adopt the file of specific compression technology, image file JPEG for example, audio files is with MP3 etc.; The one, can adopt the file of general lossless compression-encoding, as structural data library file and text or the like.
Automatically carry out by system for ease of squeeze operation, for set Database Systems, can be to each database table (that is to corresponding database file), default compress mode code is shown in following inventory.Comprise the unstructured data territory in database table, for example during large object data type LOB such as image, audio frequency, this LOB territory is the pointer that points to the LOB object, the object that LOB points in certain table space of database, can to separately default compress mode code:
| Database table | AAA | BBB | CCC | DDD | EEE | NNN | |
| The compress mode code | 05 | 01 | 00 | 03 | 00 | 12 |
Each compress mode code in the inventory, corresponding corresponding compress mode is called corresponding condensing routine.For example 00 representative is not compressed, and 01 represents the LZ compression, and 02 represents JPEG compression or the like.
Certainly, for the sake of simplicity, also can only establish in the system and not compress and two kinds of compress mode classifications of general lossless compress.Can only establish the database file inventory that do not carry out compress in the system this moment.When compressed software compresses processing to certain database file, can only judge whether it belongs to this inventory.If then will not compress; If not, then directly adopt the general lossless coding of default to compress, as shown in Figure 2.
For very big database file, also it can be divided into big as far as possible data block capacity (for example X MB) and compress, the database file after the synthetic again compression.
3, the database file after the compression send the Backup Data server stores by common backup software, and generates the corresponding database file directory on the Backup Data server.Because the database table data of schedule backup are that nature identifies with the time (as year, month, day), so the corresponding database file directory, also can form the file tree structure of being convenient to retrieve naturally.Fig. 3 is the example of the database file catalogue of a kind of tree structure of the present invention.
For the ease of User Recognition, the database table title, database file title and the database file title of having compressed, its main part can all be taked same title, only extension name difference.For example:
Database table name: AAA
Database file name: AAA.tmp
The database file name of having compressed: AAA.xxx
(xxx represents compression method title, for example Zip or the like)
Second portion: to the retrieval of the Backup Data of database filing storage.Method provided by the invention is directly to retrieve Backup Data data in server library file by application program.Specify as follows:
As previously mentioned, what store on the Backup Data server is the database file that has compressed, the database application personnel can be according to the file directory on the Backup Data server, directly the database file that has compressed that needs is carried out retrieval process is got on the client computer, and when reading in client computer, it is decompressed by the application program on the client computer.
Because in the database file that has decompressed, the Backup Data that had both comprised database table, the structural information that also comprises database table, the application program of client computer can be at first be presented at the structural information of this database file on the interface of client computer, the application personnel of database are according to this structural information input search condition, the application program of client computer just can be carried out corresponding search operaqtion according to search condition.
For example, the search condition of the structural information of the database of withdrawing the money of bank and user input is as shown in the table:
| Structural information | Name | Identification card number | Withdraw the money the date | Bank withdraws the money | The amount of money of withdrawing the money |
| Search condition | xxx | xxxxxxxx | 2004 | xxxx |
The application program of client computer is according to this search condition, can compare with it this database file whole records in 2004, the data-base recording that every and above-mentioned search condition conforms to is fully just charged to the result for retrieval file, in order to show, printing or ASSOCIATE STATISTICS.
For example Shang Mian destination file can be as shown in the table:
| Withdraw the money the date | The amount of money of withdrawing the money |
| On January 5th, 2004 | 1000 |
| On March 28th, 2004 | 1800 |
| On September 30th, 2004 | 5000 |
| Add up to | 7800 |
Similarly, also can carry out correlation inquiry to a plurality of database tables, the intermediate result that certain inquiry forms can be used as the condition of further inquiring about, or the computing that middle Query Result is correlated with etc.
For example, can retrieve respectively the database of depositing and withdrawing of same time period of bank, both subtract each other, and draw balance of deposits with the time period etc.
Certainly, in the search method of above-mentioned database back-up data, for reducing the transmission of Information capacity, also can database file be taken into client computer, and directly carry out retrieval process by the Backup Data server, or establish one in addition and carry out retrieval process with the Backup Data server application server that directly links to each other, only the retrieval process result is passed to client computer; Carrying out the data retrieved library file also can be uncompressed, obviously just need not to carry out corresponding decompression operation this moment; When not comprising the structural information of former database table in the database file, this structural information also can be read from its deposit position separately by the application program of client computer or application server.
In addition, when the database file on the Backup Data server was retrieved, the user interface that application program provided also can be same or similar with original interface when database server was retrieved, and only application program is different.
Because the Backup Data of filing storage, read-onlyly do not write, and need not a large number of users and carry out concurrent retrieval, thereby replace database retrieval by document retrieval, avoid and carry out complicated database recovery operation, should be fully feasible.
Important value of the present invention mainly comprises following several aspect:
1, the present invention can greatly improve the compressibility of Backup Data, generally can with the compressibility of database structure data by present less than 2-3 doubly, bring up to 10-30 doubly.This not only can save the investment of back-up storage greatly, and can significantly reduce the data traffic in the transmission channel.Thus, the data migtation between the server, also available under many circumstances LAN IP channel replaces fiber channel, with scsi interface equipment, replaces fibre-optic interface device, thereby reduces the cost of total system greatly.
2,, no longer need the database recovery operation of wasting time and energy very much, and can carry out in real time simply and easily to the inquiry of Backup Data.
3, because the very big raising of data compression rate, also be very beneficial for the height redundancy with failover capabilities of implementation data database data, two copies of a data library file backup for example, (RAID mirror image), thus also can greatly improve the reliability of standby system.
Claims (17)
1, the compression method of database back-up data, its main characteristic is:
Data of database backs up when compression, does not compress processing with small data piece (for example 32KB), but the backup information of earlier database table being derived forms database file, again with such database file as compression processing unit.
2, the compression method of database back-up data as claimed in claim 1, the backup information that will be derived by database table is divided into big as far as possible piece capacity (for example X MB) and compresses processing.
3, the compression method of database back-up data as claimed in claim 1, when the data of database table are field formula information, with the structural information of database table (as field name, field length, field data types and constraint condition etc.), form database file with the backup information of deriving by database table.
4, the compression method of database back-up data, its main characteristic is:
To the compression of data library file, according to its different file type, system adopts different compress modes automatically, does not for example compress Zip compression, JPEG compression etc.
5, as the compression method of claim 4 described database back-up datas, for set Database Systems, can each database table (that is corresponding database file) be preset its compress mode, and by default compress mode each database file be compressed.
6,, only establish " not compressing " and " general lossless compress " two kinds of compress modes in the system, and only establish " not compressed database lists of documents " as the compression method of claim 4 described database back-up datas.When compressed software compresses processing to certain database file, only judge whether it belongs to this inventory, if then will not compress; If not, then directly adopt the general lossless coding of default to compress.
7, the squeeze operation of claim 4, or undertaken, or undertaken by hardware by software, or combine with hardware by software and to carry out.
8, the squeeze operation of claim 5, or undertaken, or undertaken by hardware by software, or combine with hardware by software and to carry out.
9, the squeeze operation of claim 6, or undertaken, or undertaken by hardware by software, or combine with hardware by software and to carry out.
10, the search method of database back-up data, its main characteristic is:
Derive and be kept at the Backup Data in the corresponding data library file on the Backup Data server by database table, it is not imported to again in the database table of former database server and retrieve, and directly the data library file is carried out retrieval process by application program.
11, as the search method of the database back-up data of claim 10, wherein application program is the database file on the Backup Data server to be got on the client computer carry out to the retrieval of data library file.
12, as the search method of the database back-up data of claim 10, wherein application program is the database file on the Backup Data server to be taken on the application server that links to each other with the Backup Data server carry out to the retrieval of data library file.
13, as the search method of the database back-up data of claim 10, wherein application program is directly to be undertaken by the Backup Data server to the retrieval of data library file.
14, as the search method of the database back-up data of claim 10, application program provides the structural information of Relational database table in user interface, (field name, field length, field data types and constraint condition etc.), insert corresponding search condition by the user in respective field, carry out retrieval process by application program.
15, as the search method of the database back-up data of claim 10, the database table structure information that application program provided is to be included in the database file, or fetch in addition.
16, as the search method of the database back-up data of claim 10, application program provides same or analogous user interface to the retrieval of data library file with to the retrieval of database to the user.
17, as the search method of the database back-up data of claim 10, database file wherein can be that compressed or unpressed.When it compresses, then when retrieving, file decompresses earlier; When it is unpressed, then directly carry out retrieval process.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 200510064796 CN1851691A (en) | 2005-04-22 | 2005-04-22 | Database back-up data compression and search method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 200510064796 CN1851691A (en) | 2005-04-22 | 2005-04-22 | Database back-up data compression and search method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN1851691A true CN1851691A (en) | 2006-10-25 |
Family
ID=37133170
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN 200510064796 Pending CN1851691A (en) | 2005-04-22 | 2005-04-22 | Database back-up data compression and search method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN1851691A (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102685436A (en) * | 2011-12-13 | 2012-09-19 | 上海天跃科技股份有限公司 | Storage method employing scalable video coding technical design |
| CN102804168A (en) * | 2009-06-19 | 2012-11-28 | 赛贝斯股份有限公司 | Data Compression For Reducing Storage Requirements In A Database System |
| CN101867559B (en) * | 2009-04-20 | 2013-06-19 | 王金波 | Method for compressing file in instant communication and network storage system |
| CN105320669A (en) * | 2014-06-19 | 2016-02-10 | 腾讯科技（深圳）有限公司 | Method and device for data storage and method and device for data reading |
| CN105389312A (en) * | 2014-09-04 | 2016-03-09 | 上海福网信息科技有限公司 | Big data migration method and tool |
| CN107341255A (en) * | 2017-07-11 | 2017-11-10 | 北京小度信息科技有限公司 | Key assignments data processing method, device and electronic equipment |
| CN109800182A (en) * | 2019-01-18 | 2019-05-24 | 深圳忆联信息系统有限公司 | It is a kind of to reduce the data storage handling method and its system for writing amplification |
| CN117421288A (en) * | 2023-12-18 | 2024-01-19 | 云和恩墨(北京)信息技术有限公司 | Database data compression method and device |
-
2005
- 2005-04-22 CN CN 200510064796 patent/CN1851691A/en active Pending
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101867559B (en) * | 2009-04-20 | 2013-06-19 | 王金波 | Method for compressing file in instant communication and network storage system |
| CN102804168A (en) * | 2009-06-19 | 2012-11-28 | 赛贝斯股份有限公司 | Data Compression For Reducing Storage Requirements In A Database System |
| CN102804168B (en) * | 2009-06-19 | 2016-03-09 | 赛贝斯股份有限公司 | The data compression of storage demand is reduced in Database Systems |
| CN102685436A (en) * | 2011-12-13 | 2012-09-19 | 上海天跃科技股份有限公司 | Storage method employing scalable video coding technical design |
| CN105320669A (en) * | 2014-06-19 | 2016-02-10 | 腾讯科技（深圳）有限公司 | Method and device for data storage and method and device for data reading |
| CN105389312A (en) * | 2014-09-04 | 2016-03-09 | 上海福网信息科技有限公司 | Big data migration method and tool |
| CN107341255A (en) * | 2017-07-11 | 2017-11-10 | 北京小度信息科技有限公司 | Key assignments data processing method, device and electronic equipment |
| CN109800182A (en) * | 2019-01-18 | 2019-05-24 | 深圳忆联信息系统有限公司 | It is a kind of to reduce the data storage handling method and its system for writing amplification |
| CN117421288A (en) * | 2023-12-18 | 2024-01-19 | 云和恩墨(北京)信息技术有限公司 | Database data compression method and device |
| CN117421288B (en) * | 2023-12-18 | 2024-06-11 | 云和恩墨(北京)信息技术有限公司 | Database data compression method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8489553B2 (en) | Managing storage of individually accessible data units | |
| CN102945242B (en) | The management method of data, system and computer system | |
| CN103733195B (en) | Manage the storage of the data for the search based on range | |
| US20180011861A1 (en) | Managing storage of individually accessible data units | |
| CN1851691A (en) | Database back-up data compression and search method | |
| EP2545451B1 (en) | Managing storage of individually accessible data units | |
| CN1731364A (en) | Method for compression and search of database backup data | |
| AU2015258326B2 (en) | Managing storage of individually accessible data units |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C57 | Notification of unclear or unknown address | ||
| DD01 | Delivery of document by public notice |
Addressee: Wang Jinbo Document name: Notification before expiration of term |
|
| C57 | Notification of unclear or unknown address | ||
| DD01 | Delivery of document by public notice |
Addressee: Wang Jinbo Document name: Notification that Application Deemed to be Withdrawn |
|
| C02 | Deemed withdrawal of patent application after publication (patent law 2001) | ||
| WD01 | Invention patent application deemed withdrawn after publication |