CN102129458B - Method and device for storing relational database - Google Patents
Method and device for storing relational database Download PDFInfo
- Publication number
- CN102129458B CN102129458B CN 201110056705 CN201110056705A CN102129458B CN 102129458 B CN102129458 B CN 102129458B CN 201110056705 CN201110056705 CN 201110056705 CN 201110056705 A CN201110056705 A CN 201110056705A CN 102129458 B CN102129458 B CN 102129458B
- Authority
- CN
- China
- Prior art keywords
- record
- section
- major key
- table section
- minimum
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method and device for storing a relational database. The method comprises the following steps: building a control table for storing the information of a table segment, wherein the information of the table segment comprises a table segment identification as well as a minimum major key value and a maximum major key value which are stored and recorded by the table segment; when a new record is inserted into a table, inserting the new record into the memory table of the table segment of the table; when the information of the table segment is changed, correspondingly updating in the control table; when the memory occupied by all records in the memory table of the table segment reaches a predefined upper limit, storing all the records in the memory table of the table segment to a table segment data file corresponding to the table segment on a hard disk; and clearing the memory table of the table segment. According to the method and the device, the relational database can be automatically segmented and stored in lines, thereby saving the cost for storing. The storage structure of the relational database in the device is suitable to serve as the storage basis of the relational database.
Description
Technical field
The present invention relates to database technical field, relate in particular to the storage means and the device of relevant database.
Background technology
Prior art provides the method for multiple fulfillment database storage.
For example, prior art one provides a kind of method of going stored data base, like Oracle methods such as (Oracle Database Concepts).This method is stored by row the record of relevant database according to the rule of going into storage earlier earlier, and major key is set up the uniqueness index of types such as B+ tree.But prior art one exists following not enough: at first the major key index occupies certain storage space, has increased system overhead; Although some existing database has been taked to improve some query performance by the mechanism of row subregion; But whether unique this improved inspection major key expense again, because system has to check the uniqueness of major key in all subregions; Subregion is many more, and the expense of inspection is big more; Secondly because by the row storage, all row are stored in together, can not read row as required, the reading part apportion is had to read earlier whole row and is filtered out unwanted row then, has so just increased hard disk IO, has reduced performance.
Prior art two provides a kind of method and device (application number/patent No.: 200810187227) that realizes based on the relevant database of row storage.But, although prior art two has solved the problem that capable stored data base can not read row as required, still have following defective: when inserting record, press the row separate storage, and the value of each row is stored the expense when having increased storage by ordering; Owing to the row separate storage, need to connect data and connect together the original record of reconstruct to row, safeguard to connect the burden that data have also increased system.
Prior art three provides a kind of MonetDB column storage database (P.A.Boncz.: " Monet:A Next-Generation DBMS Kernel For Query-Intensive Applications ").Though the MonetDB of prior art three also is the Database Systems by the row storage; It also has the advantage that reads row as required in column data storehouse; But, still have following defective: need the ROWID of stored record when storing each row, rebuild record thereby system need connect train value through ROWID; And the normally long shaping of ROWID is longer type perhaps, has increased the expense of system like this.
Prior art four provides a kind of Google BigTable (Fay Chang etc.: Bigtable:A Distributed Storage System for Structured Data).Although Google BigTable comes stored record by RowKey ordering and segmentation,, it has introduced the notion of row families (they being Column Family); Promptly a record is by one or more row group compositions; By the storage of row family, each is listed as the corresponding data files (being Google SSTable file) of family, like this during storage; A table section (Tablet) will corresponding one or more data files (SSTable file); And the value that it is stored in row family all is the Key/Value form, is not suitable for storing relational data.In fact, BigTable is the storage system of the distributed mapping table of a multidimensional, and this storage organization is not suitable for the storage basis as relevant database.
Summary of the invention
The embodiment of the invention provides a kind of storage means of relevant database, and in order to the automatic segmentation of realizing relevant database and by the row storage, and the expense when saving storage, and storage organization is suitable for the storage basis as relevant database, and this method comprises:
Set up control table, in order to the sign of storage list section and the minimum and maximum Major key of table section institute stored record, and by the minimum Major key ordering of table section;
When new record inserts the table of relevant database:
If do not have record in the table of relevant database; The then newly-built table section that should show; New record is inserted the memory table of this table section, the sign of this table section is stored to control table, the Major key of new record is stored to control table as the minimum and the maximum Major key of this table section institute's stored record;
If existing record in the table of relevant database; Then according to the minimum Major key of all table section institute stored records in the Major key of new record and the control table; Search the table section that new record should insert, new record is inserted the memory table that this shows section, press the Major key ordering during insertion; If insert the minimum or maximum Major key that has changed this table section institute stored record, then in control table, carry out corresponding renewal;
All write down shared internal memory and reach predefined and go up in limited time in the memory table of table section:
This is shown should show on all recording storage to the hard disk in section memory table a table segment data file of section correspondence, and empty the memory table of this table section; Store by row during stored record, and keep pressing the Major key ordering; If should show the segment data file on the hard disk, the then newly-built table segment data file that is somebody's turn to do; If the existing table segment data file that is somebody's turn to do on the hard disk then merges existing record in the record in the memory table and this table segment data file when storage.
The embodiment of the invention also provides a kind of memory storage of relevant database; Automatic segmentation in order to realize relevant database is also stored by row; And the expense when saving storage, and storage organization is suitable for the storage basis as relevant database, and this device comprises:
Module is set up in control table, is used to set up control table, in order to the sign of storage list section and the minimum and maximum Major key of table section institute stored record, and by the minimum Major key ordering of table section;
The table section is set up module, is used for when new record inserts the table of relevant database: if the table of relevant database does not have record, and then newly-built table section of being somebody's turn to do table; The first record insert module is used for the memory table with new this newly-built table section of record insertion; The first control table update module, the sign that is used for the table section that this is newly-built is stored to control table, and the Major key of new record is stored to control table as the minimum and maximum Major key of this newly-built table section institute stored record;
Table segment search module; Be used for when new record inserts the table of relevant database: if the existing record of the table of relevant database; Then, search the table section that new record should insert according to the minimum Major key of all table section institute stored records in the Major key of new record and the control table; The second record insert module is used for the memory table with new this table section of searching of record insertion, presses the Major key ordering during insertion; The second control table update module is used for then in control table, carrying out corresponding renewal if insert the minimum or maximum Major key that has changed this table section institute stored record of searching;
Record unloading processing module; Being used for as the memory table of table section all writes down shared internal memory and reaches predefined and go up in limited time: show the segment data file for should table section correspondence on all recording storage to the hard disk in will show section memory table one, and empty the memory table of this table section; Store by row during stored record, and keep pressing the Major key ordering; If should show the segment data file on the hard disk, the then newly-built table segment data file that is somebody's turn to do; If the existing table segment data file that is somebody's turn to do on the hard disk then merges existing record in the record in the memory table and this table segment data file when storage.
The embodiment of the invention is compared with prior art one, need not major key is set up index, the expense in the time of can saving storage; And the embodiment of the invention can solve the problem that capable stored data base can not read row as required by the row storage;
The embodiment of the invention sorts to record with table segmenting and by Major key, compares the expense in the time of can reducing storage with prior art two; And be different from prior art two needs and connect data with the row of the separate storage reconstruct raw readings that connects together, the embodiment of the invention need not to connect data and comes the reconstruct record, can not increase to safeguard the burden that connects data;
Same, need ROWID to connect train value reconstruction record thereby be different from prior art three, the ROWID that the embodiment of the invention need not stored record rebuilds record, the burden in the time of can not increasing storage;
Compare with prior art four; The only corresponding data files of table section in the embodiment of the invention; And storage is the record of relevant database, is not the data of Key/Value form, and the embodiment of the invention is the row storage by record during storage; Be not listed as the notion of family, be suitable as the storage basis of relevant database fully.
The embodiment of the invention can improve the query performance of relevant database; And realize the storage of relevant database based on major key ordering and segmentation; Can be used as the basic storage mode of the parallel computation that realizes relevant database; The table data balancing of fragmented storage is assigned in the node of parallel database, utilizes the Map/Reduce algorithm to implement concurrent operation again.
Description of drawings
In order to be illustrated more clearly in the technical scheme in the embodiment of the invention; The accompanying drawing of required use is done to introduce simply in will describing embodiment below; Obviously, the accompanying drawing in describing below only is some embodiments of the present invention, for those of ordinary skills; Under the prerequisite of not paying creative work property, can also obtain other accompanying drawing according to these accompanying drawings.In the accompanying drawings:
Fig. 1 is the process flow diagram of the storage means of relevant database in the embodiment of the invention;
Fig. 2 is the structural representation of control table in the embodiment of the invention;
Fig. 3 records a synoptic diagram in the empty table for inserting in the embodiment of the invention;
Fig. 4 is for inserting the synoptic diagram that records situation one in the nonempty list in the embodiment of the invention;
Fig. 5 is for inserting the synoptic diagram that records situation two in the nonempty list in the embodiment of the invention;
Fig. 6 writes down the synoptic diagram that shared internal memory reaches the upper limit for inserting in the embodiment of the invention to write down in the memory table that causes the section of showing;
Fig. 7 is the form synoptic diagram of expression table segment data file in the embodiment of the invention;
Fig. 8 is the synoptic diagram of expression table section division in the embodiment of the invention;
Fig. 9 is the synoptic diagram of a projection inquiry in the embodiment of the invention;
Figure 10 is the synoptic diagram of a condition query in the embodiment of the invention;
Figure 11 is the synoptic diagram of a parallel database allocation table segment data in the embodiment of the invention;
Figure 12 is the structural drawing of the memory storage of relevant database in the embodiment of the invention.
Embodiment
For the purpose, technical scheme and the advantage that make the embodiment of the invention is clearer, the embodiment of the invention is explained further details below in conjunction with accompanying drawing.At this, illustrative examples of the present invention and explanation thereof are used to explain the present invention, but not as to qualification of the present invention.
As shown in Figure 1, the storage means flow process of relevant database can comprise in the embodiment of the invention:
Elder generation's execution in step 101: set up control table, in order to the sign of storage list section and the minimum and maximum Major key of table section institute stored record, and by the minimum Major key ordering of table section;
When new record inserts table:
If do not have record in the table, then execution in step 102: the newly-built table section that should show, with the memory table of new this table section of record insertion; Then execution in step 103: sign that will show section is stored to control table, and the Major key of new record is stored to control table as the minimum and the maximum Major key of this table section institute's stored record;
If existing record in the table; Then execution in step 104: according to the minimum Major key of all table section institute stored records in the Major key of new record and the control table; Search the table section that new record should insert, new record is inserted the memory table that this shows section, press the Major key ordering during insertion; If insert the minimum or maximum Major key that has changed this table section institute stored record, then also need execution in step 105: in control table, carry out corresponding renewal;
All write down shared internal memory and reach predefined and go up in limited time in the memory table of table section:
Execution in step 106: should a corresponding table segment data file of table section on all recording storage to the hard disk in will the section of showing memory table, by the row storage, and keep press Major key and sort during stored record; Follow execution in step 107: the memory table that empties this table section; Wherein when execution in step 106,, then also comprise: newly-built this table segment data file (step 106a) if should not show the segment data file on the hard disk; Should table segment data file if existing on the hard disk, then also comprise: when storage with the record in the memory table and this table segment data file in existing record merging (step 106b).
Do following the detailed description in the face of flow process shown in Figure 1 down:
In the embodiment of the invention, the indumentum section of being divided in the relevant database is referred to as the table section, and the record number in each table section has a upper limit.The table section is made up of memory table and table segment data file.Memory table is a core resident data structure in the table section; In order to storing the record of new insertion, and by the Major key ordering of record, the internal memory that the record of being stored takies can not surpass the predefined upper limit; Otherwise need with recording storage in table segment data file, empty memory table then.Table segment data file is stored by row in order to the record of storage list section, and presses the Major key ordering of record.Control table is stored the information of each table section in order to the admin table section, comprises the sign of table section and the minimum and maximum Major key of table section institute stored record.The table segment information that control table is stored can sort by the minimum Major key that the table section is stored.
During practical implementation, at first set up control table,, comprise the sign (ID) of table section and the minimum and maximum Major key of table section institute stored record in order to the information of storage list section.
For example, Fig. 2 is the structural representation of control table in the embodiment of the invention.As shown in Figure 2, comprise the sign (TABLET_ID) of showing section in the control table, the minimum Major key (MIN_PK) and the maximum Major key (MAX_PK) of the stored record of table section institute.When record did not insert table, control table was empty.
During practical implementation, when new record inserts table, be divided into two kinds of situation:
The one, there is not record in the table; Control table this moment is empty; Need newly-built table section that should table, new record inserted in memory table of this table section, with information stores of this table section in control table; Comprise that specifically the sign with this table section is stored to control table, the Major key of new record is stored to control table as the minimum and the maximum Major key of this table section institute stored record.Owing to have only a record in the table section, the minimum of table section institute stored record and maximum Major key are for inserting the Major key of record.
For example, Fig. 3 records a synoptic diagram in the empty table for inserting in the embodiment of the invention.As shown in Figure 3, the structure of showing in this example for (ID, NAME), insertion be recorded as (1001; China), inserted record (1001, China) in the memory table of table section; Inserted new table segment information in the control table: (1,1001,1001); Promptly the ID of table section is 1, and the minimum Major key of table section institute stored record is 1001, and the maximum Major key of storage also is 1001.
The 2nd, existing record in the table; Control table this moment is not empty; Need search the table section that new record should insert, then according to the minimum Major key of all table section institute stored records in the Major key of new record and the control table; In the memory table of the table section that finds, insert new record, press the Major key ordering during insertion.If the record that inserts has changed the minimum or maximum Major key of this table section institute stored record, then need upgrade the information of correspondence table section in the control table.Wherein, when searching the table section that new record should insert, can adopt dichotomy to search according to the minimum Major key of all table section institute stored records in the Major key of new record and the control table.
Why the reason with binary search is owing in practical application, have hundreds and thousands of and even up to ten thousand table sections; In order to improve performance; Use binary search; Because the table segment information of storing in the control table is by minimum Major key ordering, searches the object table section of insertion and can locate with binary search according to the minimum Major key of the Major key that inserts record and all table sections:
When new record inserts table, if the Major key of new record is between the minimum Major key of two table sections, the table section that then previous table section should be inserted as new record; If the Major key of new record was then shown the table section that section should be inserted as new record with first before the minimum Major key of first table section; If the Major key of new record is in the end after the minimum Major key of a table section, the table section that then last table section should be inserted as new record.
For example, Fig. 4 records the unconverted synoptic diagram of control table in the nonempty list for inserting in the embodiment of the invention.As shown in Figure 4, two table sections are arranged in the control table: (1,1001,1005) and (2,1010,1011) wherein have record in the table section 1: (1001, China), (1005, Korea S) have record: (1010, the U.S.), (1011, Canada) in the table section 2.Insert record (1003, Japan), the minimum Major key of all table section institute stored records is from small to large in the control table: 1001,1010; The result of binary search be table section 1 (because the minimum Major key of table section 1 and 2 is 1001,1010 from small to large, like this when the storage Major key is 1003 record since 1003 greater than 1001 less than 1010, show section 1 so should insert); So in table section 1, insert record (1003, Japan), inserted (1003, Japan) in the memory table of table section 1; According to the Major key order, be followed successively by: (1001, China), (1003; Japan), (1005, Korea S); Because record is inserted in the centre of table section, the minimum and the maximum Major key of the stored record of table section institute do not change, and therefore need not to change the information of this table section in the control table.
And for example, Fig. 5 records the synoptic diagram that causes in the nonempty list that control table changes for inserting in the embodiment of the invention.As shown in Figure 5, two table sections are arranged in the control table: (1,1001,1005) and (2,1010,1011) wherein have record in the table section 1: (1001, China), (1005, Korea S) have record: (1010, the U.S.), (1011, Canada) in the table section 2.Insert record (1006, India), the minimum Major key of all table section institute stored records is in the control table: 1001,1010; The result of binary search be table section 1 (because the minimum Major key of table section 1 and 2 is 1001,1010 from small to large, like this when the storage Major key is 1006 record since 1006 greater than 1001 less than 1010, show section 1 so should insert); So in table section 1, insert record (1006, India), inserted (1006, India) in the memory table of table section 1; According to the Major key order, be followed successively by: (1001, China), (1005; Korea S), (1006, India).Because record is inserted into the last of table section, variation has taken place in the maximum Major key of table section institute stored record, and the maximum Major key that upgrades first table section in the control table is 1006: (1,1001,1006).
During practical implementation, all write down shared internal memory and reach predefined and go up in limited time in the memory table of table section:
With showing a corresponding table segment data file of section on all recording storage to the hard disk in this table section memory table, then memory table is emptied to reduce the use of internal memory, this operation can be referred to as to tighten memory table; Store by row during stored record, and keep pressing the Major key ordering; If should show the segment data file on the hard disk, the then newly-built table segment data file that is somebody's turn to do; Should table segment data file if existing on the hard disk, did the operation of tightening memory table before showing, then when storage, existing record in the record in the memory table and this table segment data file is merged, still by the row storage, and keep pressing Major key and sort.
For example, Fig. 6 writes down the synoptic diagram that shared internal memory reaches the upper limit for inserting in the embodiment of the invention to write down in the memory table that causes the section of showing.As shown in Figure 6, in the memory table of table section record (1001, China) is arranged, (1003, Japan), (1005, Korea S) store record (1002, Vietnam), (1006, India) in the table segment data file of table section.Insert record (1004, Malaysia) and cause the shared internal memory of memory table to reach the predefined upper limit, for example: suppose that ID is an integer type, NAME is character types; The shared internal memory of integer is 4 bytes, and each Chinese character accounts for 2 bytes, so, inserts (1004; Malaysia) being recorded as in the memory table behind the memory table: (1001, China), (1003, Japan); (1004, Malaysia), (1005, Korea S); The shared internal memory of memory table should be: (4+2 * 2)+(4+2 * 2)+(4+2 * 4)+(4+2 * 2)=36, suppose to be limited to 32 on the predefined internal memory, and the shared internal memory of memory table reaches the upper limit so.Existing record in record in the memory table and the table segment data file is merged storage, keep the ordering of Major key, (1001,1002,1003; 1004,1005,1006), corresponding title train value is: (China, Vietnam; Japan, Malaysia, Korea S, India), the memory table of table section is cleared.
During practical implementation, the data of being stored in the table segment data file can comprise:
Train value data and metadata that record is corresponding; The train value data are ingredients of record, and such as ID train value data and NAME train value data are arranged, all train value data combinations constitute record together.During storage, ID train value data and NAME train value data separate storage.
Wherein:
The train value data number of each row equates; Because the number of record decision train value data number is so they must equate.Such as: record (1001, China) is arranged, (1002, the U.S.), the record number is 2, stores by row: ID:1001,1002; NAME: China, the U.S.; The train value data number of each row is 2, all equals the number 2 that writes down;
The train value data of each row are divided into data block by fixing train value data number, form the train value data block;
Metadata store is in meta data block; Position, record number in table segment data file and meta data block the position in table segment data file of each train value data block of metadata record in table segment data file.The number of record number=each train value data, rather than all train value data numbers, such as: record is arranged: (1001, China), (1002, the U.S.), during by the row storage, ID:1001,1002; NAME: China, the U.S.; The record number is 2, and the number of each train value data also is 2, and it is 4 that all train value data numbers are added up.
During practical implementation, the storage order of train value data and metadata can be in the table segment data file:
The train value data of first row, the train value data of second row are until the train value data and the metadata of last row.
The position of the train value data block of metadata record in table segment data file can obtain by row memory row Value Data the time, and the position of meta data block in table segment data file is last the train value data block that is right after last row.Because metadata is very little usually, can in internal memory, copy to accelerate access speed by the buffer memory portion.
For example, Fig. 7 is the form synoptic diagram of table segment data file in the embodiment of the invention.As shown in Figure 7, stored record is: (1001, China), (1002, Vietnam), (1003, Japan), (1004, Malaysia), (1005, Korea S) and (1006, India).It is 2 that division train value data become the train value data number of train value data block, and the train value data block of first row has three, is respectively: (1001,1002), (1003,1004); (1005,1006), the train value data block of second row has three, is respectively: (China, Vietnam), (Japan; Malaysia), (Korea S, India), meta data block is: (0,8,16; 32,48,68,6,84); Wherein, 0 is the position of train value data block (1001,1002) in table segment data file, and 8 is the position of train value data block (1003,1004) in table segment data file; 16 is the position of train value data block (1005,1006) in table segment data file, and 32 is the position of train value data block (China, Vietnam) in table segment data file, and 48 is train value data block (Japan; Malaysia) position in table segment data file, 68 is the position of train value data block (Korea S, India) in table segment data file, and 6 are the record number, and 84 is the position of meta data block in table segment data file.
During practical implementation, reach predefined as all record counts that the table section is stored and go up in limited time: partial record is retained in this table section, and newly-built table section, with all the other recording storage to newly-built table section; For example, the first half record is retained in this table section, and newly-built table section, with after half recording storage newly-built table section extremely.Because the maximum Major key of this table section changes, in control table, upgrade the maximum Major key of this table section; Minimum and the maximum Major key of the sign of newly-increased newly-built table section and institute's stored record in control table is pressed the minimum Major key ordering of table section during insertion.
For example, Fig. 8 is the synoptic diagram of table section division in the embodiment of the invention.As shown in Figure 8, stored record is in the table section: (1001, China), (1002, Vietnam), (1003; Japan), (1004, Malaysia), (1005, Korea S) and (1006, India); Inserting record (1007, the Hong-Kong), to cause the section of showing to reach the dominant record of table section several 7, and the his-and-hers watches section divides, and becomes two table sections: table section 1 (1,1001,1004) with show sections 2 (2; 1005,1007), wherein show being recorded as of section 1 storage: (1001, China), (1002; Vietnam), (1003, Japan), (1004, Malaysia); And being recorded as of table section 2 storages: (1005, Korea S), (1006, India), (1007; The Hong-Kong), the maximum Major key of table section 1 changes 1004 into from 1006 in the control table, and increases the information of table section 2 newly: the table segment identification is 2, and minimum Major key is 1005, and maximum Major key is 1007.
Following brief account is by some basic database query operations of the relevant database of embodiment of the invention method storage.
One, projection inquiry
The projection inquiry is meant some row that writes down in the question blank; Because the embodiment of the invention is by the row stored record; So just can read row as required; And the database that kind that is different from the row storage need read the data of all row, has so just significantly reduced the hard disk input and output, thereby has improved the query performance of relevant database.
For example, Fig. 9 is the synoptic diagram of a projection inquiry in the embodiment of the invention.As shown in Figure 9, the projection query statement is: " SELECT NAME FROM COUNTRY ", because the embodiment of the invention is pressed row stored record data; Can read-onlyly the fall in lines data of NAME; The output result, and the Value Data of row ID is not read, thus improved query performance.
Two, condition query
Condition query is meant that query statement has querying condition; Because the embodiment of the invention is by the row stored record; Can read the record of relevant row according to querying condition, in case the train value of record has passed through these querying conditions, the every other train value of output that needs just is read; Output in the middle of the final result, reduced IO than line data storehouse like this.
For example, Figure 10 is the synoptic diagram of a condition query in the embodiment of the invention.Shown in figure 10; The statement of condition query is: " SELECT ID FROM COUNTRY WHERE NAME=' Korea S " '; Value Data that can at first sharp scan columns NAME during inquiry is found the 5th record and is satisfied condition, and reads the value of the row ID of the 5th record like this and since in the embodiment of the invention in each train value data block access the train value data of fixed number are arranged; Train value data number like each train value data block access among Figure 10 is 2; Can calculate qualified ID value like this in the 3rd data block, then, can utilize the position data of the train value data block of storing in the metadata to locate the 3rd data block of row ID fast; Thereby read qualified ID value, promptly 1005.
Three, parallel processing
Because the indumentum in the embodiment of the invention is divided into the table section, like this, if in the environment of a cluster and cloud computing; Because the existence of a plurality of servers is arranged, can give each server with the uniform distribution of table section, during record in the question blank; Server in the cluster just can launch computing simultaneously; Deal with the record data in the own hard disk separately, reach the purpose of parallel processing, this just can improve performance of database greatly.
For example, Figure 11 is the synoptic diagram of a Cluster Database parallel processing in the embodiment of the invention.Shown in figure 11, Cluster Database is made up of two servers, and table is made up of two table sections, is stored in respectively in two servers, and each server provides the record queries service of relevant table section.
A kind of memory storage of relevant database also is provided in the embodiment of the invention, of following embodiment.Because the principle of this device solves problem is similar with the storage means of relevant database, so the enforcement of this device can be referring to the enforcement of the storage means of relevant database, and the repetition part repeats no more.
Shown in figure 12, the memory storage of relevant database can comprise in the embodiment of the invention:
Table section is set up module 1202, is used for when new record insertion table: if table does not have record, and then newly-built table section that should table; The first record insert module 1203 is used for the memory table with new this newly-built table section of record insertion; The first control table update module 1204, the sign that is used for the table section that this is newly-built is stored to control table, and the Major key of new record is stored to control table as the minimum and maximum Major key of this newly-built table section institute stored record;
Table segment search module 1205 is used for when new record insertion table: if the existing record of table, then according to the minimum Major key of all table section institute stored records in the Major key of new record and the control table, search the table section that new record should insert; The second record insert module 1206 is used for the memory table with new this table section of searching of record insertion, presses the Major key ordering during insertion; The second control table update module 1207 is used for then in control table, carrying out corresponding renewal if insert the minimum or maximum Major key that has changed this table section institute stored record of searching;
Record unloading processing module 1208; Being used for as the memory table of table section all writes down shared internal memory and reaches predefined and go up in limited time: show the segment data file for should table section correspondence on all recording storage to the hard disk in will show section memory table one, and empty the memory table of this table section; Store by row during stored record, and keep pressing the Major key ordering; If should show the segment data file on the hard disk, the then newly-built table segment data file that is somebody's turn to do; If the existing table segment data file that is somebody's turn to do on the hard disk then merges existing record in the record in the memory table and this table segment data file when storage.
Among the embodiment, table segment search module 1205 specifically can be used for: according to the Major key of new record and the minimum Major key of all table section institute stored records of control table, and the table section that adopts the new record of binary search to insert:
If the Major key of new record is between the minimum Major key of two table sections, the table section that then previous table section should be inserted as new record;
If the Major key of new record was then shown the table section that section should be inserted as new record with first before the minimum Major key of first table section;
If the Major key of new record is in the end after the minimum Major key of a table section, the table section that then last table section should be inserted as new record.
Among the embodiment, the data of being stored in the table segment data file comprise train value data and the metadata that record is corresponding; Wherein:
The train value data number of each row equates; The train value data of each row are divided into data block by fixing train value data number, form the train value data block;
Metadata store is in meta data block; Position, record number in table segment data file and meta data block the position in table segment data file of each train value data block of metadata record in table segment data file.
Among the embodiment, the storage order of train value data and metadata is in the table segment data file:
The train value data of first row, the train value data of second row are until the train value data and the metadata of last row.
Among the embodiment, metadata cache is in internal memory.
Among the embodiment, the memory storage of relevant database shown in Figure 12 can also comprise:
The table section splits module, is used for reaching predefined as all record counts that the table section is stored and goes up in limited time: partial record is retained in this table section, and newly-built table section, with all the other recording storage to newly-built table section;
The 3rd control update module is used for then in control table, carrying out corresponding renewal if the minimum or maximum Major key of this table section institute stored record changes; Minimum and the maximum Major key of the sign of newly-increased newly-built table section and institute's stored record in control table is pressed the minimum Major key ordering of table section during insertion.
Among the embodiment, table section splits a module and specifically can be used for: the first half record is retained in this table section, and newly-built table section, with after half recording storage newly-built table section extremely.
The embodiment of the invention is compared with prior art one, need not major key is set up index, can save system overhead; And the embodiment of the invention can solve the problem that capable stored data base can not read row as required by the row storage;
The embodiment of the invention sorts to record with table segmenting and by Major key, compares the expense in the time of can reducing storage with prior art two; And be different from prior art two needs and connect data with the row of the separate storage reconstruct raw readings that connects together, the embodiment of the invention need not to connect data and comes the reconstruct record, can not increase to safeguard the system burden that connects data;
Same, need ROWID to connect train value reconstruction record thereby be different from prior art three, the ROWID that the embodiment of the invention need not stored record rebuilds record, can not increase system burden;
Compare with prior art four; The only corresponding data files of table section in the embodiment of the invention; And storage is the record of relevant database, is not the data of Key/Value form, and the embodiment of the invention is the row storage by record during storage; Be not listed as the notion of family, be suitable as the storage basis of relevant database fully.
The embodiment of the invention can improve the query performance of database; And realize the storage of relevant database based on major key ordering and segmentation; Can be used as the basic storage mode of the parallel computation that realizes relevant database; The table data balancing of fragmented storage is assigned in the node of parallel database, utilizes the Map/Reduce algorithm to implement concurrent operation again.
Those skilled in the art should understand that embodiments of the invention can be provided as method, system or computer program.Therefore, the present invention can adopt the form of the embodiment of complete hardware embodiment, complete software implementation example or combination software and hardware aspect.And the present invention can be employed in the form that one or more computer-usable storage medium (including but not limited to magnetic disk memory, CD-ROM, optical memory etc.) that wherein include computer usable program code go up the computer program of implementing.
The present invention is that reference is described according to the process flow diagram and/or the block scheme of method, equipment (system) and the computer program of the embodiment of the invention.Should understand can be by the flow process in each flow process in computer program instructions realization flow figure and/or the block scheme and/or square frame and process flow diagram and/or the block scheme and/or the combination of square frame.Can provide these computer program instructions to the processor of multi-purpose computer, special purpose computer, Embedded Processor or other programmable data processing device to produce a machine, make the instruction of carrying out through the processor of computing machine or other programmable data processing device produce to be used for the device of the function that is implemented in flow process of process flow diagram or a plurality of flow process and/or square frame of block scheme or a plurality of square frame appointments.
These computer program instructions also can be stored in ability vectoring computer or the computer-readable memory of other programmable data processing device with ad hoc fashion work; Make the instruction that is stored in this computer-readable memory produce the manufacture that comprises command device, this command device is implemented in the function of appointment in flow process of process flow diagram or a plurality of flow process and/or square frame of block scheme or a plurality of square frame.
These computer program instructions also can be loaded on computing machine or other programmable data processing device; Make on computing machine or other programmable devices and to carry out the sequence of operations step producing computer implemented processing, thereby the instruction of on computing machine or other programmable devices, carrying out is provided for being implemented in the step of the function of appointment in flow process of process flow diagram or a plurality of flow process and/or square frame of block scheme or a plurality of square frame.
Above-described specific embodiment; The object of the invention, technical scheme and beneficial effect have been carried out further explain, and institute it should be understood that the above is merely specific embodiment of the present invention; And be not used in qualification protection scope of the present invention; All within spirit of the present invention and principle, any modification of being made, be equal to replacement, improvement etc., all should be included within protection scope of the present invention.
Claims (14)
1. the storage means of a relevant database is characterized in that, comprising:
Set up control table, in order to the sign of storage list section and the minimum and maximum Major key of table section institute stored record, and by the minimum Major key ordering of table section;
When new record inserts the table of relevant database:
If do not have record in the table of relevant database; The then newly-built table section that should show; New record is inserted the memory table of this table section, the sign of this table section is stored to control table, the Major key of new record is stored to control table as the minimum and the maximum Major key of this table section institute's stored record;
If existing record in the table of relevant database; Then according to the minimum Major key of all table section institute stored records in the Major key of new record and the control table; Search the table section that new record should insert, new record is inserted the memory table that this shows section, press the Major key ordering during insertion; If insert the minimum or maximum Major key that has changed this table section institute stored record, then in control table, carry out corresponding renewal;
All write down shared internal memory and reach predefined and go up in limited time in the memory table of table section:
This is shown should show on all recording storage to the hard disk in section memory table a table segment data file of section correspondence, and empty the memory table of this table section; Store by row during stored record, and keep pressing the Major key ordering; If should show the segment data file on the hard disk, the then newly-built table segment data file that is somebody's turn to do; If the existing table segment data file that is somebody's turn to do on the hard disk then merges existing record in the record in the memory table and this table segment data file when storage.
2. the method for claim 1 is characterized in that, the minimum Major key of all table section institute stored records is confirmed the table section that new record should insert in the Major key of the record that said basis is new and the control table, comprising:
According to the minimum Major key of all table section institute stored records in the Major key of new record and the control table, the table section that adopts the new record of binary search to insert:
If the Major key of new record is between the minimum Major key of two table sections, the table section that then previous table section should be inserted as new record;
If the Major key of new record was then shown the table section that section should be inserted as new record with first before the minimum Major key of first table section;
If the Major key of new record is in the end after the minimum Major key of a table section, the table section that then last table section should be inserted as new record.
3. the method for claim 1 is characterized in that, the data of being stored in the table segment data file comprise train value data and the metadata that record is corresponding; Wherein:
The train value data number of each row equates; The train value data of each row are divided into data block by fixing train value data number, form the train value data block;
Metadata store is in meta data block; Position, record number in table segment data file and meta data block the position in table segment data file of each train value data block of metadata record in table segment data file.
4. method as claimed in claim 3 is characterized in that, the storage order of train value data and metadata is in the table segment data file:
The train value data of first row, the train value data of second row are until the train value data and the metadata of last row.
5. method as claimed in claim 3 is characterized in that metadata cache is in internal memory.
6. the method for claim 1 is characterized in that, also comprises:
Reaching predefined as table section all record counts of being stored goes up in limited time:
Partial record is retained in this table section, and newly-built table section, with all the other recording storage to newly-built table section;
If the minimum or maximum Major key of this table section institute stored record changes, then in control table, carry out corresponding renewal;
Minimum and the maximum Major key of the sign of newly-increased newly-built table section and institute's stored record in control table is pressed minimum Major key ordering during insertion.
7. method as claimed in claim 6 is characterized in that, said partial record is retained in this table section, and newly-built table section, with all the other recording storage newly-built table section extremely, comprising:
The first half record is retained in this table section, and newly-built table section, with after half recording storage newly-built table section extremely.
8. the memory storage of a relevant database is characterized in that, comprising:
Module is set up in control table, is used to set up control table, in order to the sign of storage list section and the minimum and maximum Major key of table section institute stored record, and by the minimum Major key ordering of table section;
The table section is set up module, is used for when new record inserts the table of relevant database: if the table of relevant database does not have record, and then newly-built table section of being somebody's turn to do table; The first record insert module is used for the memory table with new this newly-built table section of record insertion; The first control table update module, the sign that is used for the table section that this is newly-built is stored to control table, and the Major key of new record is stored to control table as the minimum and maximum Major key of this newly-built table section institute stored record;
Table segment search module; Be used for when new record inserts the table of relevant database: if the existing record of the table of relevant database; Then, search the table section that new record should insert according to the minimum Major key of all table section institute stored records in the Major key of new record and the control table; The second record insert module is used for the memory table with new this table section of searching of record insertion, presses the Major key ordering during insertion; The second control table update module is used for then in control table, carrying out corresponding renewal if insert the minimum or maximum Major key that has changed this table section institute stored record of searching;
Record unloading processing module; Being used for as the memory table of table section all writes down shared internal memory and reaches predefined and go up in limited time: show the segment data file for should table section correspondence on all recording storage to the hard disk in will show section memory table one, and empty the memory table of this table section; Store by row during stored record, and keep pressing the Major key ordering; If should show the segment data file on the hard disk, the then newly-built table segment data file that is somebody's turn to do; If the existing table segment data file that is somebody's turn to do on the hard disk then merges existing record in the record in the memory table and this table segment data file when storage.
9. device as claimed in claim 8 is characterized in that, said table segment search module specifically is used for: according to the Major key of new record and the minimum Major key of all table section institute stored records of control table, and the table section that adopts the new record of binary search to insert:
If the Major key of new record is between the minimum Major key of two table sections, the table section that then previous table section should be inserted as new record;
If the Major key of new record was then shown the table section that section should be inserted as new record with first before the minimum Major key of first table section;
If the Major key of new record is in the end after the minimum Major key of a table section, the table section that then last table section should be inserted as new record.
10. device as claimed in claim 8 is characterized in that, the data of being stored in the table segment data file comprise train value data and the metadata that record is corresponding; Wherein:
The train value data number of each row equates; The train value data of each row are divided into data block by fixing train value data number, form the train value data block;
Metadata store is in meta data block; Position, record number in table segment data file and meta data block the position in table segment data file of each train value data block of metadata record in table segment data file.
11. device as claimed in claim 10 is characterized in that, the storage order of train value data and metadata is in the table segment data file:
The train value data of first row, the train value data of second row are until the train value data and the metadata of last row.
12. device as claimed in claim 10 is characterized in that metadata cache is in internal memory.
13. device as claimed in claim 8 is characterized in that, also comprises:
The table section splits module, is used for reaching predefined as all record counts that the table section is stored and goes up in limited time: partial record is retained in this table section, and newly-built table section, with all the other recording storage to newly-built table section;
The 3rd control update module is used for then in control table, carrying out corresponding renewal if the minimum or maximum Major key of this table section institute stored record changes; Minimum and the maximum Major key of the sign of newly-increased newly-built table section and institute's stored record in control table is pressed minimum Major key ordering during insertion.
14. device as claimed in claim 13 is characterized in that, said table section splits a module and specifically is used for: the first half record is retained in this table section, and newly-built table section, with after half recording storage newly-built table section extremely.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201110056705 CN102129458B (en) | 2011-03-09 | 2011-03-09 | Method and device for storing relational database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201110056705 CN102129458B (en) | 2011-03-09 | 2011-03-09 | Method and device for storing relational database |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102129458A CN102129458A (en) | 2011-07-20 |
| CN102129458B true CN102129458B (en) | 2012-12-12 |
Family
ID=44267540
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN 201110056705 Active CN102129458B (en) | 2011-03-09 | 2011-03-09 | Method and device for storing relational database |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102129458B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104424258A (en) * | 2013-08-28 | 2015-03-18 | 腾讯科技（深圳）有限公司 | Multidimensional data query method and system, query server and column storage server |
Families Citing this family (25)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102291268B (en) * | 2011-09-23 | 2014-11-26 | 杜跃进 | Safety domain name server and hostile domain name monitoring system and method based on same |
| CN103218365A (en) * | 2012-01-20 | 2013-07-24 | 阿里巴巴集团控股有限公司 | SS Table file data processing method and system |
| CN102682108B (en) * | 2012-05-08 | 2015-02-18 | 同方知网数字出版技术股份有限公司 | Row and line mixed database storage method |
| CN102768672B (en) * | 2012-06-12 | 2016-02-17 | 北大方正集团有限公司 | A kind of disk space management method and apparatus |
| CN103870483B (en) * | 2012-12-13 | 2018-04-20 | 厦门雅迅网络股份有限公司 | A kind of method that dynamic adjustment memory headroom batch stores data |
| CN103914462B (en) * | 2012-12-31 | 2017-09-05 | 中国移动通信集团公司 | A kind of data storage and query method and device |
| CN103246498A (en) * | 2013-05-13 | 2013-08-14 | 浪潮集团山东通用软件有限公司 | Memory storage structures supporting relational data parallel processing and achieving method thereof |
| CN104252486B (en) * | 2013-06-28 | 2017-09-12 | 阿里巴巴集团控股有限公司 | A kind of method and device of data processing |
| US9996568B2 (en) | 2013-12-31 | 2018-06-12 | International Business Machines Corporation | Index maintenance based on a comparison of rebuild vs. update |
| CN103812877B (en) * | 2014-03-12 | 2016-10-12 | 西安电子科技大学 | Data compression method based on Bigtable distributed memory system |
| CN104268272A (en) * | 2014-10-14 | 2015-01-07 | 青岛海尔软件有限公司 | Method and device for checking uniqueness of data |
| CN104657513B (en) * | 2015-03-20 | 2018-02-09 | 山东威尔数据股份有限公司 | Archives operation and method for quickly retrieving in embedded system |
| CN104715039B (en) * | 2015-03-23 | 2018-10-19 | 星环信息科技（上海）有限公司 | Based on the storage of the column of hard disk and memory and querying method and equipment |
| CN105302889B (en) * | 2015-10-16 | 2019-06-04 | 北京奇虎科技有限公司 | The conversion method and device of data store organisation |
| CN106570129A (en) * | 2016-10-27 | 2017-04-19 | 南京邮电大学 | Storage system for rapidly analyzing real-time data and storage method thereof |
| CN108629196B (en) * | 2017-03-21 | 2021-04-30 | 北京京东尚科信息技术有限公司 | Data storage and query method and device, electronic equipment and readable storage medium |
| CN109388636A (en) * | 2017-08-11 | 2019-02-26 | 中国移动通信集团重庆有限公司 | Business datum is inserted into database method, apparatus, computer equipment and storage medium |
| CN110019191A (en) * | 2017-09-21 | 2019-07-16 | 阿里巴巴集团控股有限公司 | Database information processing method and processing device |
| CN109033271B (en) * | 2018-07-10 | 2021-03-02 | 上海达梦数据库有限公司 | Data insertion method and device based on column storage, server and storage medium |
| CN108984720B (en) * | 2018-07-10 | 2021-06-22 | 上海达梦数据库有限公司 | Data query method and device based on column storage, server and storage medium |
| CN108984719B (en) * | 2018-07-10 | 2021-08-03 | 上海达梦数据库有限公司 | Data deleting method and device based on column storage, server and storage medium |
| CN108875077B (en) * | 2018-07-10 | 2021-02-09 | 上海达梦数据库有限公司 | Column storage method and device of database, server and storage medium |
| CN108959587B (en) * | 2018-07-10 | 2021-03-02 | 上海达梦数据库有限公司 | Data updating method and device based on column storage, server and storage medium |
| CN109522269B (en) * | 2018-09-30 | 2023-04-21 | 中国农业大学烟台研究院 | File management method |
| CN109710619B (en) * | 2018-12-29 | 2023-09-29 | 中国银联股份有限公司 | Database capacity expansion method, device and readable medium |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101751406B (en) * | 2008-12-18 | 2012-01-04 | 赵伟 | Method and device for realizing column storage based relational database |
| US8108400B2 (en) * | 2009-06-27 | 2012-01-31 | Hewlett-Packard Development Company, L.P. | Database segment searching |
-
2011
- 2011-03-09 CN CN 201110056705 patent/CN102129458B/en active Active
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104424258A (en) * | 2013-08-28 | 2015-03-18 | 腾讯科技（深圳）有限公司 | Multidimensional data query method and system, query server and column storage server |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102129458A (en) | 2011-07-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102129458B (en) | Method and device for storing relational database | |
| CN101751406B (en) | Method and device for realizing column storage based relational database | |
| US10311048B2 (en) | Full and partial materialization of data from an in-memory array to an on-disk page structure | |
| US11169978B2 (en) | Distributed pipeline optimization for data preparation | |
| CA2723731C (en) | Managing storage of individually accessible data units | |
| Larson et al. | Enhancements to SQL server column stores | |
| CN105117417A (en) | Read-optimized memory database Trie tree index method | |
| US20120303633A1 (en) | Systems and methods for querying column oriented databases | |
| US20120221523A1 (en) | Database Backup and Restore with Integrated Index Reorganization | |
| CN110825748A (en) | High-performance and easily-expandable key value storage method utilizing differential index mechanism | |
| US20110213775A1 (en) | Database Table Look-up | |
| US10255234B2 (en) | Method for storing data elements in a database | |
| US10642814B2 (en) | Signature-based cache optimization for data preparation | |
| TW201530328A (en) | Method and device for constructing NoSQL database index for semi-structured data | |
| US20150058352A1 (en) | Thin database indexing | |
| US10963440B2 (en) | Fast incremental column store data loading | |
| US11288287B2 (en) | Methods and apparatus to partition a database | |
| US10884998B2 (en) | Method for migrating data records from a source database to a target database | |
| CN105677915A (en) | Distributed service data access method based on engine | |
| AU2018345147B2 (en) | Database processing device, group map file production method, and recording medium | |
| CN104408128A (en) | Read optimization method for asynchronously updating indexes based on B+ tree | |
| CN104462080A (en) | Index structure creating method and system with group statistics for search results | |
| US20240220470A1 (en) | Data storage device and storage control method based on log-structured merge tree | |
| US8073823B2 (en) | Database management program | |
| CN1492363A (en) | Data storage and searching method of embedded system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| ASS | Succession or assignment of patent right |
Owner name: BEIJING HANYUN TIMES TECHNOLOGY CO., LTD. Free format text: FORMER OWNER: HU JINSONG Effective date: 20120221 |
|
| C41 | Transfer of patent application or patent right or utility model | ||
| COR | Change of bibliographic data |
Free format text: CORRECT: ADDRESS; FROM: 247000 CHIZHOU, ANHUI PROVINCE TO: 100142 HAIDIAN, BEIJING |
|
| TA01 | Transfer of patent application right |
Effective date of registration: 20120221 Address after: 100142, 608, Fu Cheng Road, Beijing, Beijing, Haidian District, room 58, Haidian District Applicant after: Beijing Cloudwave Times Technology Co., Ltd. Address before: 247000 Anhui city of Chizhou province Cuiwei court E District 12 building 201 room Applicant before: Hu Jinsong |
|
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| C56 | Change in the name or address of the patentee | ||
| CP01 | Change in the name or title of a patent holder |
Address after: 100142 room 58, Fu Cheng Road, 608, Beijing, Haidian District Patentee after: Beijing Han Yun era Data Technology Co. Ltd. Address before: 100142 room 58, Fu Cheng Road, 608, Beijing, Haidian District Patentee before: Beijing Cloudwave Times Technology Co., Ltd. |