CN106682134A - Method, device and system for writing data pairs in LevelDB databases - Google Patents
Method, device and system for writing data pairs in LevelDB databases Download PDFInfo
- Publication number
- CN106682134A CN106682134A CN201611168789.7A CN201611168789A CN106682134A CN 106682134 A CN106682134 A CN 106682134A CN 201611168789 A CN201611168789 A CN 201611168789A CN 106682134 A CN106682134 A CN 106682134A
- Authority
- CN
- China
- Prior art keywords
- data
- leveldb
- databases
- message queue
- writing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2308—Concurrency control
- G06F16/2315—Optimistic concurrency control
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Security & Cryptography (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the technical field of computer data storage, in particular to a method, device and system for writing data pairs in LevelDB databases. The method provided herein comprises: configuring multiple LevelDB databases corresponding to multiple message queues; writing data pairs into one message queue; writing the data pairs of the message queue into the LevelDB database corresponding to the message queue, wherein before configuring the multiple databases corresponding to the multiple message queues, steps also are included which include: creating multiple catalogues, and configuring multiple catalogues corresponding to the multiple LevelDB databases, and wherein writing the data pairs into one message queue includes: allocating the data pairs among the multiple message queues through Hash algorithm, and determining the message queue which the data pairs are to be written in. The problem that writing performance of LevelDB databases under high stress is low is solved, and the writing performance of LevelDB databases is improved.
Description
Technical field
The present invention relates to data in technical field of computer data storage, more particularly to a kind of leveldb databases to writing
Method, the apparatus and system for entering.
Background technology
Leveldb databases are the key-value storage engines storehouses that google increases income, and are previously mentioned in present specification
Data to being key-value pairs, leveldb databases in data storage, be according to record key values in order store,
That is adjacent key values are sequential storage successively in storage file, and applying can compare letter with self-defined key sizes
Number, leveldb databases can sequentially store these records according to user-defined comparison function.
In field of data storage, the application scenarios of leveldb databases are a lot, in the application process of leveldb databases
Can there is a problem of that some performances are not high：Under such as large batch of write-in scene, the files of level-0 in levedb databases
Quantity reaches kL0_SlowdownWritesTrigger（Acquiescence 8）When, produce write latency problem；Reach kL0_
StopWritesTrigger（Acquiescence 12）When, write-in obstructing problem is produced, result in the overall write-in of leveldb databases
Can be relatively low.
Conventional solution is kL0_SlowdownWritesTrigger and kL0_ in modification source code at present
The size of StopWritesTrigger come adjust write latency and write-in obstruction condition, but, the program can not be solved thoroughly
The certainly problem.Because under the scene of high-volume write-in, by improving the value of two above variable, temporarily can only extend to reach and write
Enter to postpone the time with write-in congestion condition, and do not settle the matter once and for all.After reaching the time, can still trigger and write
Enter to postpone the condition with write-in obstruction, cause write performance low.This scheme can only extend to the time up to trigger condition, and
Write performance is low after can not solving the problems, such as to reach trigger condition.
The content of the invention
In consideration of it, being solved to the method, the apparatus and system that write process provides data in a kind of leveldb databases
Determine the relatively low problem of the write performance of leveldb databases under big pressure condition, improve the write-in of leveldb databases
Performance.
In order to achieve the above object, the present invention is achieved by the following technical solutions：
The present invention provides method of the data to writing in a kind of leveldb databases, including：
The multiple message queue correspondence multiple leveldb databases of configuration；
By data to writing one of message queue；
By the data in message queue to writing the leveldb database corresponding with the message queue.
Wherein, before configuring multiple message queue correspondence multiple databases, also including step：Create multiple catalogues, configuration
Multiple leveldb databases correspondence multiple catalogues.
Wherein, before by data to the one of message queue of write-in, including step：By hash algorithm by data pair
It is allocated between multiple message queues, determines data to the message queue that will write.
Preferably, by asynchronous thread by data to being written to message queue.
Wherein, by the data in message queue to before writing the leveldb databases corresponding with the message queue, going back
Including step：Thread pool is created in leveldb databases, thread pool includes multiple threads.
Wherein, by the data in message queue to writing the leveldb database corresponding with the message queue, including step
Suddenly：Multiple threads are by the data in message queue to writing the leveldb database corresponding with the message queue.
The present invention provides a kind of device of data in leveldb databases to writing, including：
Message queue configuration module, for configuring the multiple message queues corresponding with multiple leveldb databases；
First data to writing module, for by data to write-in message queue；
Second data to writing module, for the data in message queue are corresponding with the message queue to writing
Leveldb databases.
Preferably, also include：
Directory creating module, for creating multiple catalogues；
Leveldb database configuration modules, for configuring the multiple leveldb databases being associated with multiple catalogues.
Preferably, also include：
Hash operation module, for carrying out Hash operation, determining data to the message queue that will write to data.
The present invention provides system of the data to writing in a kind of leveldb databases, including：Data are to arithmetic element, different
Step thread and multiple catalogues, wherein, the directory name of multiple catalogues is different, is respectively equipped with each catalogue corresponding with the catalogue
Leveldb databases, message queue and thread pool, the Data Computation Unit to data to carrying out Hash operation after, pass through
Asynchronous thread writes data into the message queue under one of catalogue, and the thread pool includes multiple threads, and thread will disappear
Data in breath queue are to writing the leveldb database corresponding with the message queue.
Beneficial effects of the present invention are as follows：
Method of the data to writing, has the following advantages that in a kind of leveldb databases provided by the present invention：
1. by creating multiple catalogues, and multiple leveldb databases are configured under multiple catalogues, when data are to write-in, when
When one of database occurs write latency or write-in obstruction, being normally written for other databases is had no effect on, while will be
Under the scene of high-volume write-in, the integral pressure dispersion of data write-in gives multiple databases, and write-in is reduced from another point of view
Postpone or write the occurrence probability of obstruction, improve data write efficiency and performance；
2. hash algorithm is taken by the key to data centering, realize equiblibrium mass distribution of the data to storage, quick decision data pair
Which database will be fallen into, the write efficiency of database will be improved from one side；
3. after determining data to the database to be fallen into by hash algorithm, data are carried out by asynchronous thread to disappearing
The write-in of queue is ceased, the write operation speed of asynchronous thread is far above synchronism thread, therefore reduces leveldb database processings
Thread latency during write operation, improves treatment effeciency；
4. by way of multithread pool, thread pool is contained within multiple threads, using multiple threading operations multiple data pair, carries
Data high are to reaching the efficiency of database；
In leveldb databases data to write apparatus and system beneficial effect with data in leveldb databases to writing
The beneficial effect of the method for entering is similar to, and repeats no more.
Brief description of the drawings：
The schematic flow sheet of method of the data to writing in the leveldb databases that Fig. 1 is provided by the embodiment of the present invention；
The schematic diagram of device of the data to writing in the leveldb databases that Fig. 2 is provided by the embodiment of the present invention；
The frame diagram of system of the data to writing in the leveldb databases that Fig. 3 is provided by the embodiment of the present invention；
Specific embodiment：
Core of the invention is to provide data in a kind of leveldb databases and, to the method, the apparatus and system that write, solves
The relatively low problem of the write performance of leveldb databases, improves the write performance of leveldb databases under big pressure condition.
To make the purpose, technical scheme and advantage of the embodiment of the present invention clearer, below in conjunction with the embodiment of the present invention
In accompanying drawing, the technical scheme in the embodiment of the present invention is clearly and completely described, it is clear that described embodiment is
A part of embodiment of the present invention, rather than whole embodiments.Based on the embodiment in the present invention, those of ordinary skill in the art
The every other embodiment obtained under the premise of creative work is not made, belongs to the scope of protection of the invention.
Refer to Fig. 1, method of the data to writing in the leveldb databases that Fig. 1 is provided by the embodiment of the present invention
Schematic flow sheet；Data include to the method for writing in leveldb databases：
Step S101, the multiple catalogues of establishment, configuration multiple leveldb database correspondence multiple catalogues；
Wherein, multiple catalogues are created on disk, the quantity of catalogue is designated as db_num, by user's sets itself or can be
System default setting, because the quantity of catalogue is a definite value, therefore each catalogue is different from other equivalent to one
The directory name of catalogue, then for each catalogue configures a leveldb database db, the quantity of database is also db_num, often
One database has unique name, is designated as dbID.
It is that catalogue configuration leveldb databases can be by creating a new leveldb as a kind of embodiment
Database is opened existing leveldb database and is realized.
Step S102, the multiple message queue correspondence multiple leveldb databases of configuration；
Wherein, the quantity db_num according to database come be each database configure a message queue, the number of message queue
Amount is also db_num, therefore the dbID of data is applied equally to corresponding message queue.
Step S103, by hash algorithm by data to being allocated between multiple message queues, determine data to will
The message queue to be write；
Wherein, the data described in present specification are to referring to contain key（Key）、value（Value）Data, data to write-in
When message queue queue [dbID], it is necessary to carry out Hash operation to key after, remainder is carried out to db_num to obtain the number
According to database dbID, formula is：dbID = hash（key） % db_num.For example：Data to for（17, value）, number
It it is 5 according to the quantity in storehouse, their dbID is followed successively by 1,2,3,4,5, i.e. key=17, db_num=5, if Hash operation is direct
Addressing, i.e. hash（key）=key, then be not difficult to draw dbID=2, i.e. data pair（17, value）Message queue will be written into
queue[2]。
Used as a kind of embodiment, Hash operation can select digital analysis method, middle square method, jackknife method, random
Number method, leaving remainder method etc..
Step S104, by asynchronous thread by data to being written to message queue.Asynchronous thread is by data pair（Key,
value）Write-in message queue queue [dbID], then asynchronous thread return processes other message.
Step S105, thread pool is created in leveldb databases, thread pool includes multiple threads；In leveldb numbers
According to storehouse db [dbID] thread pools Tpool [dbID], multiple threads are included in thread pool Tpool [dbID].
Step S106, by the data in message queue to writing the leveldb database corresponding with the message queue；
Wherein, multiple threads by the data in message queue to writing the leveldb database corresponding with the message queue.Should
The data pair in thread process message queue queue [dbID] in the corresponding thread pool Tpool [dbID] of database（Key,
value）, it is written into leveldb databases db [dbID].
Refer to Fig. 2, data are to the device that writes in the leveldb databases that Fig. 2 is provided by the embodiment of the present invention
Schematic diagram；Data include to the device for writing in leveldb databases：
Message queue configuration module 203, for configuring the multiple message queues corresponding with multiple leveldb databases；
First data to writing module 204, for by data, to write-in message queue, asynchronous thread to be by data pair（Key,
value）Write-in message queue queue [dbID], then asynchronous thread return processes other message；
Second data to writing module 205, for the data in message queue are corresponding with the message queue to writing
Leveldb databases.Thread process message queue in the corresponding thread pool Tpool [dbID] of current leveldb databases
Data pair in queue [dbID]（Key, value）, it is written into leveldb databases db [dbID].
Directory creating module 201, for creating multiple catalogues；
Leveldb databases configuration module 202, for configuring the multiple leveldb databases being associated with multiple catalogues.
Hash operation module 206, for carrying out Hash operation, determining data to the message team that will write to data
Row.
Refer to Fig. 3, system of the data to writing in the leveldb databases that Fig. 3 is provided by the embodiment of the present invention
Frame diagram；Data include to the system for writing in leveldb databases：
Data to arithmetic element, asynchronous thread and multiple catalogue, wherein, the directory name of multiple catalogues is different, each catalogue
Under be respectively equipped with leveldb databases corresponding with the catalogue, message queue and thread pool, the Data Computation Unit is to data
After carrying out Hash operation, the message queue under one of catalogue, the thread pool bag are write data into by asynchronous thread
Multiple threads are included, thread is by the data in message queue to writing the leveldb database corresponding with the message queue.
Schematical specific embodiment of the invention is the foregoing is only, the scope of the present invention is not limited to, it is any
The equivalent variations that those skilled in the art is made on the premise of present inventive concept and principle is not departed from and modification, all should belong to
In the scope of protection of the invention.
Claims (10)
1. in a kind of leveldb databases data to write method, it is characterised in that including：
The multiple message queue correspondence multiple leveldb databases of configuration；
By data to writing one of message queue；
By the data in message queue to writing the leveldb database corresponding with the message queue.
2. method of the data to writing in leveldb databases according to claim 1, it is characterised in that configuration is multiple
Before message queue correspondence multiple leveldb databases, including：Create multiple catalogues, configuration multiple leveldb database correspondences
Multiple catalogues.
3. in leveldb databases according to claim 1 data to write method, it is characterised in that by data pair
Before writing one of message queue, including：By hash algorithm by data to being divided between multiple message queues
Match somebody with somebody, determine data to the message queue that will write.
4. in leveldb databases according to claim 1 data to write method, it is characterised in that by data pair
One of message queue is write, including：By asynchronous thread by data to being written to message queue.
5. in leveldb databases according to claim 1 data to write method, it is characterised in that by message team
Data in row are to before writing the leveldb databases corresponding with the message queue, also including：In leveldb databases
Middle establishment thread pool, thread pool includes multiple threads.
6. in leveldb databases according to claim 5 data to write method, it is characterised in that by message team
Data in row to writing the leveldb database corresponding with the message queue, including：Multiple threads are by message queue
Data are to writing the leveldb database corresponding with the message queue.
7. in a kind of leveldb databases data to write device, it is characterised in that including：
Message queue configuration module, for configuring the multiple message queues corresponding with multiple leveldb databases；
First data to writing module, for by data to write-in message queue；
Second data to writing module, for the data in message queue are corresponding with the message queue to writing
Leveldb databases.
8. in leveldb databases according to claim 7 data to write device, it is characterised in that also include：
Directory creating module, for creating multiple catalogues；
Leveldb database configuration modules, for configuring the multiple leveldb databases being associated with multiple catalogues.
9. in leveldb databases according to claim 7 data to write device, it is characterised in that also include：
Hash operation module, for carrying out Hash operation, determining data to the message queue that will write to data.
10. in a kind of leveldb databases data to write system, it is characterised in that including：Data are to arithmetic element, different
Step thread and multiple catalogues, wherein, the directory name of multiple catalogues is different, is respectively equipped with each catalogue corresponding with the catalogue
Leveldb databases, message queue and thread pool, the Data Computation Unit to data to carrying out Hash operation after, pass through
Asynchronous thread writes data into the message queue under one of catalogue, and the thread pool includes multiple threads, and thread will disappear
Data in breath queue are to writing the leveldb database corresponding with the message queue.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611168789.7A CN106682134A (en) | 2016-12-16 | 2016-12-16 | Method, device and system for writing data pairs in LevelDB databases |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611168789.7A CN106682134A (en) | 2016-12-16 | 2016-12-16 | Method, device and system for writing data pairs in LevelDB databases |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106682134A true CN106682134A (en) | 2017-05-17 |
Family
ID=58869616
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611168789.7A Pending CN106682134A (en) | 2016-12-16 | 2016-12-16 | Method, device and system for writing data pairs in LevelDB databases |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106682134A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110019345A (en) * | 2017-12-28 | 2019-07-16 | 北京京东尚科信息技术有限公司 | Data processing method, device, system and medium |
| CN110766854A (en) * | 2019-10-30 | 2020-02-07 | 口碑(上海)信息技术有限公司 | Voting data processing method and device |
| CN115629890A (en) * | 2022-10-27 | 2023-01-20 | 圣名科技(广州)有限责任公司 | Sensor data acquisition method and device and electronic equipment |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102521265A (en) * | 2011-11-21 | 2012-06-27 | 华中科技大学 | Dynamic consistency control method in massive data management |
| CN102622426A (en) * | 2012-02-27 | 2012-08-01 | 杭州闪亮科技有限公司 | Database writing system and database writing method |
| CN103268318A (en) * | 2013-04-16 | 2013-08-28 | 华中科技大学 | Distributed key value database system with strong consistency and read-write method thereof |
| CN105224445A (en) * | 2015-10-28 | 2016-01-06 | 北京汇商融通信息技术有限公司 | Distributed tracking system |
-
2016
- 2016-12-16 CN CN201611168789.7A patent/CN106682134A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102521265A (en) * | 2011-11-21 | 2012-06-27 | 华中科技大学 | Dynamic consistency control method in massive data management |
| CN102622426A (en) * | 2012-02-27 | 2012-08-01 | 杭州闪亮科技有限公司 | Database writing system and database writing method |
| CN103268318A (en) * | 2013-04-16 | 2013-08-28 | 华中科技大学 | Distributed key value database system with strong consistency and read-write method thereof |
| CN105224445A (en) * | 2015-10-28 | 2016-01-06 | 北京汇商融通信息技术有限公司 | Distributed tracking system |
Non-Patent Citations (1)
| Title |
|---|
| 秦峥惠: "基于Leveldb的企业级大数据集群化存储设计与实现", 《中国优秀硕士学位论文全文库 信息科技辑》 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110019345A (en) * | 2017-12-28 | 2019-07-16 | 北京京东尚科信息技术有限公司 | Data processing method, device, system and medium |
| CN110766854A (en) * | 2019-10-30 | 2020-02-07 | 口碑(上海)信息技术有限公司 | Voting data processing method and device |
| CN115629890A (en) * | 2022-10-27 | 2023-01-20 | 圣名科技(广州)有限责任公司 | Sensor data acquisition method and device and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2020233212A1 (en) | Log record processing method, server, and storage medium | |
| WO2020224023A1 (en) | Reshard method and system in distributed storage system | |
| CN105204781B (en) | Compression method, device and equipment | |
| WO2019033605A1 (en) | Method and apparatus for asynchronous order processing, and storage medium and terminal | |
| WO2021018199A1 (en) | Convolutional neural network-based image processing method and apparatus | |
| CN106790170B (en) | Data packet filtering method and device | |
| CN106682004A (en) | Redis Key management method and system | |
| WO2021052169A1 (en) | Equalization processing method and device for distributed data, computing terminal and storage medium | |
| CN104881466B (en) | The processing of data fragmentation and the delet method of garbage files and device | |
| CN106682134A (en) | Method, device and system for writing data pairs in LevelDB databases | |
| WO2014037767A1 (en) | Multi-level inline data deduplication | |
| CN107391770B (en) | Method, device and equipment for processing data and storage medium | |
| CN106383742B (en) | A kind of I O scheduling method based on linux | |
| CN102456076A (en) | Massive fragment data aggregation system and method | |
| CN103392169A (en) | Sorting | |
| US20190114303A1 (en) | System and method for applying extended regular expressions against arbitrary data objects | |
| CN103701469A (en) | Compression and storage method for large-scale image data | |
| EP3933743A1 (en) | Method and device for blockchain transaction tracing | |
| CN104572505A (en) | System and method for ensuring eventual consistency of mass data caches | |
| WO2021027612A1 (en) | Method and apparatus for executing transaction in blockchain | |
| CN109471843A (en) | A kind of metadata cache method, system and relevant apparatus | |
| US20220253222A1 (en) | Data reduction method, apparatus, computing device, and storage medium | |
| CN112162975A (en) | Method for realizing repeated data deletion technology based on single-hash equal-distribution bloom filter | |
| CN105068875A (en) | Intelligence data processing method and apparatus | |
| WO2021004266A1 (en) | Data insertion method and apparatus, device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170517 |
|
| RJ01 | Rejection of invention patent application after publication |