CN104809178A - Write-in method of key/value database memory log - Google Patents
Write-in method of key/value database memory log Download PDFInfo
- Publication number
- CN104809178A CN104809178A CN201510178361.XA CN201510178361A CN104809178A CN 104809178 A CN104809178 A CN 104809178A CN 201510178361 A CN201510178361 A CN 201510178361A CN 104809178 A CN104809178 A CN 104809178A
- Authority
- CN
- China
- Prior art keywords
- record
- file
- log
- key
- value database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a write-in method of a key/value database memory log. The method comprises the following steps that according to the number N of disks capable of being practically used by a system, N log files are written in the disks in parallel, and in addition, each file in the N log files is put in different disks. According to the method, the key/value database log file is split into a plurality of sub files by adopting a Hash algorithm, the multi-disc characteristics are sufficiently utilized, and meanwhile, the sub files are written into different disks in parallel, so that the I/O (input/output) seek time is effectively reduced, the write-in concurrency of the key/value database log is improved, and the fast and efficient log write-in is realized.
Description
Technical field
The application relates to calculator memory management domain, relates more specifically to the wiring method of a kind of key value database internal memory daily record.
Background technology
Common database system uses WAL (Write Ahead Logging) write-in policy, does not namely directly write in database file during Update Table, but is first written in journal file (Log file); If affairs failure, the record in daily record can be left in the basket, and cancels amendment; If affairs success, it will be written back in database file in certain time subsequently, submits amendment to.
Log file is as the daily record of Database Systems, its principle of work is: database before write memory first by operation note in Log file, and then write memory, even if system crash like this, also can recover corresponding data from Log file, thus avoid loss of data.If do not have Log file, because the record of write has just started to be kept in internal memory, if now system crash, the data in internal memory have not also had enough time to be stored into disk, will restart or the reason such as power-off and losing because of internal memory.
In practical operation, because all write operations all need first to write daily record, the bottleneck place be written in order to whole system write performance of daily record, therefore how solving daily record write bottleneck becomes the key improving throughput of system and concurrency.
Summary of the invention
In view of this, one of fundamental purpose of the present invention is the wiring method proposing the daily record of a kind of key value database internal memory, to utilize the characteristic of hash algorithm and multiple-disk system, improves daily record write concurrency, effective reduction IO seek time, realizes daily record write rapidly and efficiently.
To achieve these goals, the present invention proposes the wiring method of a kind of key value database internal memory daily record, comprise the following steps:
According to the actual disk number N that can use of system, be written in parallel to N number of Log file, and each in described N number of Log file is placed on different disks respectively;
Wherein, N is natural number.
Wherein, described be written in parallel to N number of Log file before first by hash algorithm, original Log file declustering is obtained M Log son file, described M Log son file preserves into identical suffix name file, if and M > N, then be written in parallel to N number of son file at every turn, remain being written in parallel to for the last time less than N, be written in parallel to until all files has all write by repeatedly such; If M≤N, be then once written in parallel to M file;
Wherein, M is natural number.
Wherein, described Log file is made up of three blocks.
Wherein, the size of each of described three blocks is 32K.
Wherein, described Log file is key-value pair sequence.
Wherein, described Log file comprises multiple key-value pair, and wherein each key-value pair is a record, and each described record comprises a record-header, to be used for recording management information.
Wherein, described record-header comprises three fields: CRC32 field, record-length field and record type field; Wherein, described CRC32 field is used for verifying record type and data field, and described record-length field is for recording the size of data, and described record type field is for identifying the relation between the logical organization of every bar record and physical block structure.
Wherein, described record type field has four kinds of values: FULL, FIRST, MIDDLE, LAST; Wherein, FULL type shows that this record contains complete data; When key-value pair size has exceeded every block 32K, need to store data in many records, now the record type of Article 1 record has been set to FIRST, and the record type of intermediate record is set to MIDDLE, and the record type of the last item record is set to LAST.
Wherein, the shortest size of described record is 7B, i.e. the size of record-header.
Wherein, when the remaining space of described Log blocks of files is less than 6B, remainder will be filled to be null character string.
Known based on technique scheme, method of the present invention improves concurrency by Hash hashing algorithm, multiple hard disk writes simultaneously, reduce random IO, improve handling capacity, thus reach Appropriate application hardware resource, effectively reduce IO seek time, finally reach high concurrent, height is handled up, daily record write rapidly and efficiently.
Accompanying drawing explanation
Fig. 1 is Log file block structured schematic layout pattern;
Fig. 2 is the structural representation of record in database.
Embodiment
For making the object, technical solutions and advantages of the present invention clearly understand, below in conjunction with specific embodiment, and with reference to accompanying drawing, the present invention is described in further detail.
Journal file (Log file) Main Function is in systems in which the data that prevent loss when system crash recovers; because write operations all in prior art all need first to write daily record, the bottleneck place be written in order to whole system write performance of therefore daily record.
In order to improve the writing speed of journal file, the present invention attempts by hash algorithm, and the operation of write journal file resolved into and write multiple file simultaneously, but find when reality uses, the write efficiency of database does not improve, and reduces greatly on the contrary.Known by analysis, journal file, after Hash, can be written in different files.During write Different L og file, disk calls random disk I/O usually, and random disk I/O is non-conterminous due to twice writing address, and likely wide apart, magnetic head needs the Hen Changhuan road time, and magnetic head ceaselessly changes and causes system performance greatly to reduce.
In order to avoid the drawback that above-mentioned random disk I/O brings, polylith disk is organized by the present invention, first by hash algorithm, journal file is divided into the son file with disk number equal number, then each Log son file is placed on different disks respectively, thus avoids random disk I/O.Its implementation is also very simple, only needs catalogues different for each disk allocation, if any three pieces of disks, respectively the position storing Log in three pieces of disks is flexible coupling for/home/Deluge/Log0 ,/home/Deluge/Log1 ,/home/Deluge/Log2.Write in three file :/home/Deluge/Log0/logging0.XXX ,/home/Deluge/Log1/logging1.XXX ,/home/Deluge/Log2/logging1.XXX after being recorded in Hash respectively, this can cause being sequentially written in of three disks, and can not cause the generation of random disk I/O.Each Log son file preserves into identical suffix name file, for the mark as reduction log content.
Below in conjunction with accompanying drawing, further explanation is explained to a preferred embodiment of the present invention.
the physical layout of Log file
Database of the present invention is the reading of Log and write conveniently, Log file is split in units of 32K, using block (chunk) as the unit of disk read-write, the Log file that Fig. 1 shows is made up of three blocks, so from physical layout, Log file is exactly the combination of a series of block in units of 32K.
the logic placement of Log file
In logic, database by abstract for Log file be key-value pair sequence.In a database, a key-value pair is regarded as record data, before data, be provided with a record-header in addition, be used for recording management information, to facilitate inter-process.Fig. 2 shows the composition structure recorded in database of the present invention.
Article one, record is made up of record-header and data.Wherein, data refer to the key-value pair needing write into Databasce.It is check codes to record type and data field that record-header comprises three field: CRC32, and the object of this field is in order to avoid processing imperfect or destroyed data.Database can carry out CRC32 verification to data when reading record data, if result is identical with CRC32 field, then illustrates that data are complete and correct, can continue follow-up flow process.Otherwise, abandon this record.Record-length field have recorded the size of data.Record type field identifies the relation between the logical organization of every bar record and physical block structure, and this field has four kinds of values: FULL, FIRST, MIDDLE, LAST.FULL type shows that this record contains complete data; In fact because the size of key-value pair may be larger, exceed the size restriction of every block 32K, just need to store data in many records, Article 1, the record type of record is set to FIRST, the record type of intermediate record is set to MIDDLE, and the record type of the last item record is set to LAST.As shown in Figure 2, because the size recording B has exceeded 32k, so be stored in multiple pieces by record B, and each piece has independently record-header, and according to the position difference that each Log records, arranges its record type.
Because the shortest size of record is 7B, i.e. the size of record-header.If when the remaining space of Log block is less than 6B, remainder will be filled to be null character string.
When being written in disk by Log file, system is first by hash algorithm, and Log file declustering is become M file, wherein M is natural number.Secondly, system, according to the characteristic of many disks, according to the actual disk number N (N is natural number) that can use of system, write operation is resolved into and is written in parallel to N number of Log son file simultaneously, wherein, if M > is N, then after being written in parallel to N number of file, be again written in parallel to remaining file, be written in parallel to until all files has all write by repeatedly such; If M≤N, be then written in parallel to M file.In the process being written in parallel to N/M sub-Log file to each disk, each in this N/M Log son file is placed on different disks respectively, daily record write concurrency can be improved thus, effectively reduce IO seek time, realize daily record write rapidly and efficiently.
The hash algorithm used in the present invention can be hashing algorithm well known in the art, as long as reversibly can restore original Log file from Log son file.
Above-described specific embodiment; object of the present invention, technical scheme and beneficial effect are further described; be understood that; the foregoing is only specific embodiments of the invention; be not limited to the present invention; within the spirit and principles in the present invention all, any amendment made, equivalent replacement, improvement etc., all should be included within protection scope of the present invention.
Claims (10)
1. a wiring method for key value database internal memory daily record, comprises the following steps:
According to the actual disk number N that can use of system, be written in parallel to N number of Log file, and each in described N number of Log file is placed on different disks respectively;
Wherein, N is natural number.
2. the wiring method of key value database internal memory daily record as claimed in claim 1, wherein described be written in parallel to N number of Log file before first by hash algorithm, original Log file declustering is obtained M Log son file, described M Log son file preserves into identical suffix name file, if and M > N, then be written in parallel to N number of son file at every turn, remain being written in parallel to for the last time less than N, be written in parallel to until all files has all write by repeatedly such; If M≤N, be then once written in parallel to M file;
Wherein, M is natural number.
3. the wiring method of key value database internal memory daily record as claimed in claim 1, wherein said Log file is made up of three blocks.
4. the wiring method of key value database internal memory daily record as claimed in claim 3, the size of each of wherein said three blocks is 32K.
5. the wiring method of key value database internal memory daily record as claimed in claim 1, wherein said Log file is key-value pair sequence.
6. the wiring method of key value database internal memory daily record as claimed in claim 5, wherein said Log file comprises multiple key-value pair, and wherein each key-value pair is a record, and each described record comprises a record-header, to be used for recording management information.
7. the wiring method of key value database internal memory daily record as claimed in claim 6, wherein said record-header comprises three fields: CRC32 field, record-length field and record type field; Wherein, described CRC32 field is used for verifying record type and data field, and described record-length field is for recording the size of data, and described record type field is for identifying the relation between the logical organization of every bar record and physical block structure.
8. the wiring method of key value database internal memory daily record as claimed in claim 7, wherein said record type field has four kinds of values: FULL, FIRST, MIDDLE, LAST; Wherein, FULL type shows that this record contains complete data; When key-value pair size has exceeded every block 32K, need to store data in many records, now the record type of Article 1 record has been set to FIRST, and the record type of intermediate record is set to MIDDLE, and the record type of the last item record is set to LAST.
9. the wiring method of key value database internal memory daily record as claimed in claim 6, the shortest size of wherein said record is 7B, i.e. the size of record-header.
10. the wiring method of key value database internal memory daily record as claimed in claim 6, wherein when the remaining space of described Log blocks of files is less than 6B, remainder will be filled to be null character string.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510178361.XA CN104809178A (en) | 2015-04-15 | 2015-04-15 | Write-in method of key/value database memory log |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510178361.XA CN104809178A (en) | 2015-04-15 | 2015-04-15 | Write-in method of key/value database memory log |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN104809178A true CN104809178A (en) | 2015-07-29 |
Family
ID=53694000
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510178361.XA Pending CN104809178A (en) | 2015-04-15 | 2015-04-15 | Write-in method of key/value database memory log |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104809178A (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105068765A (en) * | 2015-08-13 | 2015-11-18 | 浪潮(北京)电子信息产业有限公司 | Log processing method and system based on key value database |
| CN106776901A (en) * | 2016-11-30 | 2017-05-31 | 北京知道创宇信息技术有限公司 | Data extraction method, apparatus and system |
| CN106886375A (en) * | 2017-03-27 | 2017-06-23 | 百度在线网络技术（北京）有限公司 | The method and apparatus of data storage |
| CN108984566A (en) * | 2017-06-02 | 2018-12-11 | 伊姆西Ip控股有限责任公司 | Method and apparatus for file system journal |
| KR20190021773A (en) * | 2017-08-23 | 2019-03-06 | 삼성전자주식회사 | Electronic device and method for operating data of the same |
| WO2019127456A1 (en) * | 2017-12-29 | 2019-07-04 | 华为技术有限公司 | Method and apparatus for processing write ahead log |
| CN111061690A (en) * | 2019-11-22 | 2020-04-24 | 武汉达梦数据库有限公司 | RAC-based database log file reading method and device |
| CN111104373A (en) * | 2019-12-24 | 2020-05-05 | 天地伟业技术有限公司 | Database performance optimization method |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050262170A1 (en) * | 2004-05-19 | 2005-11-24 | Oracle International Corporation | Real-time apply mechanism in standby database environments |
| US20110218971A1 (en) * | 2010-03-08 | 2011-09-08 | Yahoo! Inc. | System, Method And Computer Program Product For Managing Caches |
| CN102810050A (en) * | 2011-05-31 | 2012-12-05 | 深圳市金蝶友商电子商务服务有限公司 | Log data writing method and log system |
| CN103092903A (en) * | 2011-11-07 | 2013-05-08 | Sap股份公司 | Database Log Parallelization |
| CN103729442A (en) * | 2013-12-30 | 2014-04-16 | 华为技术有限公司 | Method for recording event logs and database engine |
| CN104281506A (en) * | 2014-07-10 | 2015-01-14 | 中国科学院计算技术研究所 | Data maintenance method and system for file system |
-
2015
- 2015-04-15 CN CN201510178361.XA patent/CN104809178A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050262170A1 (en) * | 2004-05-19 | 2005-11-24 | Oracle International Corporation | Real-time apply mechanism in standby database environments |
| US20110218971A1 (en) * | 2010-03-08 | 2011-09-08 | Yahoo! Inc. | System, Method And Computer Program Product For Managing Caches |
| CN102810050A (en) * | 2011-05-31 | 2012-12-05 | 深圳市金蝶友商电子商务服务有限公司 | Log data writing method and log system |
| CN103092903A (en) * | 2011-11-07 | 2013-05-08 | Sap股份公司 | Database Log Parallelization |
| CN103729442A (en) * | 2013-12-30 | 2014-04-16 | 华为技术有限公司 | Method for recording event logs and database engine |
| CN104281506A (en) * | 2014-07-10 | 2015-01-14 | 中国科学院计算技术研究所 | Data maintenance method and system for file system |
Cited By (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105068765A (en) * | 2015-08-13 | 2015-11-18 | 浪潮(北京)电子信息产业有限公司 | Log processing method and system based on key value database |
| CN106776901A (en) * | 2016-11-30 | 2017-05-31 | 北京知道创宇信息技术有限公司 | Data extraction method, apparatus and system |
| CN106776901B (en) * | 2016-11-30 | 2019-12-06 | 北京知道创宇信息技术股份有限公司 | Data extraction method, device and system |
| CN106886375B (en) * | 2017-03-27 | 2019-11-05 | 百度在线网络技术（北京）有限公司 | The method and apparatus of storing data |
| CN106886375A (en) * | 2017-03-27 | 2017-06-23 | 百度在线网络技术（北京）有限公司 | The method and apparatus of data storage |
| CN108984566A (en) * | 2017-06-02 | 2018-12-11 | 伊姆西Ip控股有限责任公司 | Method and apparatus for file system journal |
| KR20190021773A (en) * | 2017-08-23 | 2019-03-06 | 삼성전자주식회사 | Electronic device and method for operating data of the same |
| KR102405593B1 (en) | 2017-08-23 | 2022-06-08 | 삼성전자 주식회사 | Electronic device and method for operating data of the same |
| US11379458B2 (en) | 2017-08-23 | 2022-07-05 | Samsung Electronics Co., Ltd | Electronic device and data management method thereof |
| CN110402436A (en) * | 2017-12-29 | 2019-11-01 | 华为技术有限公司 | The method and apparatus for handling write-ahead log |
| WO2019127456A1 (en) * | 2017-12-29 | 2019-07-04 | 华为技术有限公司 | Method and apparatus for processing write ahead log |
| CN110402436B (en) * | 2017-12-29 | 2021-10-08 | 华为技术有限公司 | Method and device for processing pre-written log |
| US11526495B2 (en) | 2017-12-29 | 2022-12-13 | Huawei Technologies Co., Ltd. | Method and apparatus for processing write-ahead log |
| CN111061690A (en) * | 2019-11-22 | 2020-04-24 | 武汉达梦数据库有限公司 | RAC-based database log file reading method and device |
| CN111061690B (en) * | 2019-11-22 | 2023-08-22 | 武汉达梦数据库股份有限公司 | RAC-based database log file reading method and device |
| CN111104373A (en) * | 2019-12-24 | 2020-05-05 | 天地伟业技术有限公司 | Database performance optimization method |
| CN111104373B (en) * | 2019-12-24 | 2023-09-19 | 天地伟业技术有限公司 | Database performance optimization method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104809178A (en) | Write-in method of key/value database memory log | |
| CN105573681B (en) | Method and system for establishing RAID in SSD | |
| US10365983B1 (en) | Repairing raid systems at per-stripe granularity | |
| CN103136243B (en) | File system duplicate removal method based on cloud storage and device | |
| CN105843551B (en) | Data integrity and loss resistance in high performance and large capacity storage deduplication | |
| CN103729442B (en) | Record the method and database engine of transaction journal | |
| CN100541453C (en) | Large capacity cache implementation method and storage system | |
| CN107391544B (en) | Processing method, device and equipment of column type storage data and computer storage medium | |
| CN103064765A (en) | Method and device for data recovery and cluster storage system | |
| CN103559027A (en) | Design method of separate-storage type key-value storage system | |
| US10467074B2 (en) | Conditional journal for storage class memory devices | |
| CN109407985B (en) | Data management method and related device | |
| CN103019884A (en) | Memory page de-weight method and memory page de-weight device based on virtual machine snapshot | |
| CN104765574A (en) | Data cloud storage method | |
| CN104462388B (en) | A kind of redundant data method for cleaning based on tandem type storage medium | |
| CN103729149A (en) | Data storage method | |
| CN107205024A (en) | Data duplicate removal method and system in a kind of cloud storage system | |
| CN103370113A (en) | Data storage method and data storage system | |
| CN106909554A (en) | A kind of loading method and device of database text table data | |
| CN105302665A (en) | Improved copy-on-write snapshot method and system | |
| US11093339B2 (en) | Storage utilizing a distributed cache chain and a checkpoint drive in response to a data drive corruption | |
| CN111444114B (en) | Method, device and system for processing data in nonvolatile memory | |
| CN104778100A (en) | Safe data backup method | |
| CN113918089A (en) | Key value storage device and method for sorting key values | |
| CN103412802B (en) | Disaster tolerant data file accesses the method and device controlling list backup |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| EXSB | Decision made by sipo to initiate substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| AD01 | Patent right deemed abandoned | ||
| AD01 | Patent right deemed abandoned |
Effective date of abandoning: 20181102 |