CN105550271A - Memory database query method and device based on key value pairs - Google Patents
Memory database query method and device based on key value pairs Download PDFInfo
- Publication number
- CN105550271A CN105550271A CN201510907956.4A CN201510907956A CN105550271A CN 105550271 A CN105550271 A CN 105550271A CN 201510907956 A CN201510907956 A CN 201510907956A CN 105550271 A CN105550271 A CN 105550271A
- Authority
- CN
- China
- Prior art keywords
- data
- query
- block
- district
- memory database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2246—Trees, e.g. B+trees
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
Abstract
The invention relates to the data query field, especially a memory database query method and device based on key value pairs. The memory database query method and device provided by the invention aims at the problems of the prior art. All non- query operations of the memory database are managed through an interface thread; real time data are sorted through a data updating thread; the data are output to a data preparation block for querying the data; the data in a data operation buffer are updated to the real time data; then the data are exchanged; and the data are switched after the data are prepared well. Meanwhile, the query interface is set to only access to the data of a data query block, therefore, the purpose of the invention is realized. The starting thread of the invention receives a data operation command and the data through a Socket and writes the data operation command and the data in a data operation buffer; the query interface carries out dichotomy query in a data query area according to the pointed direction of the pointer at the head of the data query area and the key values; and a query result is returned by the query interface.
Description
Technical field
The present invention relates to data query field, especially a kind of memory database querying method based on key-value pair and device.
Background technology
Under project application environment, we need the database of a high frequency queries millions, and require that average lookup time is within 20us.Therefore, conventional database can not satisfy the demands; And the conventional memory data structure for inquiring about: ordered list, RB-Tree, AVL-Tree etc., when inserting, deleting data, also can produce serious interference to high-frequency inquiry, cause system can not be operated in good state.Therefore, we need one: 1 can bear high frequency inquiry, 2 can return Query Result at a high speed, and 3 can support ten million order of magnitude, 4 can not affect when data manipulation normal queries operation memory database.
Summary of the invention
Technical matters to be solved by this invention is: for prior art Problems existing, a kind of memory database querying method based on key-value pair and device are provided, the present invention is separated by data separating, operation, the mode of key assignments inquiry, and in conjunction with the advantage of different pieces of information structure, organising data.Particular by all non-query manipulation of this memory database of interface line thread management; The data encasement block of data query is outputted to, by the Data Update of data manipulation buffer zone to real time data by the sequence of Data Update thread real time data; Then management data query block, data encasement block, and after data ready, realize switching between the two.Query interface only visit data query block data are set simultaneously, reach object of the present invention.
The technical solution used in the present invention is as follows:
A kind of memory database querying method based on key-value pair comprises:
Step 1: with random value by after Hash algorithm for encryption, query key will be formed, and query key and described character string structure are formed a key-value pair;
Step 2: memory database maintenance module arranges the data manipulation buffer zone that memory database maintenance module application memory size can be arranged; Construct a RB_Tree as real time data district; Apply for that one piece of shared drive is as data query district, and data query district head, data query block pointer, data encasement block pointer, data query block, data encasement block are set for data query district;
Step 3: memory database maintenance module opens the Data Update thread for the interface thread and data real-time update carrying out command operating; Described unlatching thread receives data operation commands and data by Socket, and by data manipulation instruction and data write data manipulation buffer zone;
Step 4: client call query interface, query interface, according to the pointed of data query district head, carries out dichotomy inquiry according to key-value pair, and returns Query Result by query interface in data query district.
Further, in described step 3, the Data Update thread specific works process of data real-time update is:
Step 31: memory database maintenance module turn-on data more new thread, described Data Update thread, by the data of data manipulation buffer zone, adds real time data district to by data manipulation instruction;
Step 32: after having added data, Data Update thread is by real time data district data sorting and output to data encasement block;
Step 33: after data output to data encasement block completely, Data Update thread exchanges and points to the pointer of data query block and the pointer of data encasement block; The described pointer of sensing data query block and the pointer of data encasement block are arranged in data query build portion (reaching the object exchanging data query block and data encasement block), perform step 31.
A kind of memory database inquiry unit based on key-value pair comprises:
Key-value pair forms module, for random value by after Hash algorithm for encryption, forming query key, and query key and described character string structure are formed a key-value pair;
Memory database maintenance module, for arranging the data manipulation buffer zone that memory database maintenance module application memory size can be arranged; Construct a RB_Tree as real time data district; Apply for that one piece of shared drive is as data query district, and data query district head, data query block pointer, data encasement block pointer, data query block, data encasement block are set for data query district; Then memory database maintenance module opens the Data Update thread for the interface thread and data real-time update carrying out command operating; Described unlatching thread receives data operation commands and data by Socket, and by data manipulation instruction and data write data manipulation buffer zone;
Client, for calling query interface, query interface, according to the pointed of data query district head, carries out dichotomy inquiry according to key-value pair, and returns Query Result by query interface in data query district.
Further, the Data Update thread specific works process of described data real-time update is:
Step 31: memory database maintenance module turn-on data more new thread, described Data Update thread, by the data of data manipulation buffer zone, adds real time data district to by data manipulation instruction;
Step 32: after having added data, Data Update thread is by real time data district data sorting and output to data encasement block;
Step 33: after data output to data encasement block completely, Data Update thread exchanges and points to the pointer of data query block and the pointer of data encasement block; The described pointer of sensing data query block and the pointer of data encasement block are arranged in data query build portion (reaching the object exchanging data query block and data encasement block), perform step 31.
In sum, owing to have employed technique scheme, the invention has the beneficial effects as follows:
1, be separated by data separating, operation, the mode of key assignments inquiry, and in conjunction with the advantage of different pieces of information structure, organising data.
2, query manipulation is separated with non-query manipulation, and the non-query manipulation of database is by interface line thread management, and query manipulation can carry out at any time, and without any need for mutual exclusion action.
3, real time data is separated with data query.Data query block, data encasement block are separated, and by interface line thread management, external inquiry is without any perception.
The advantage of the present invention compared with existing database, data query structure is that can bear high frequency inquires about, and can return Query Result at a high speed, can support ten million order of magnitude, can not affect the ability of the memory database of normal queries operation when data manipulation;
Through actual measurement, when 2,000 ten thousand character string, 1,000 ten thousand concurrent inquiries, average lookup time only needs 7us, the poorest about 10us, optimum approximately 3us.
Accompanying drawing explanation
Examples of the present invention will be described by way of reference to the accompanying drawings, wherein:
Fig. 1 is the invention process process.
Embodiment
All features disclosed in this instructions, or the step in disclosed all methods or process, except mutually exclusive feature and/or step, all can combine by any way.
Arbitrary feature disclosed in this instructions (comprising any accessory claim, summary and accompanying drawing), unless specifically stated otherwise, all can be replaced by other equivalences or the alternative features with similar object.That is, unless specifically stated otherwise, each feature is an example in a series of equivalence or similar characteristics.
1, random value refers to arbitrary string, numeral;
2, data manipulation instruction refers to the instructions such as interpolation, deletion or amendment;
3, memory database maintenance module can be realized by a process;
4, data separating, operation separation, key assignments inquiry.Utilize the advantage of different pieces of information structure, organising data.By allowing certain data syn-chronization time delay, the entirety realizing memory database is efficient.
5, DATA POPULATION is divided into 3 parts: data manipulation buffer zone, real time data 1 district, data query district.Data query is divided into again: data query block, data encasement block.
The course of work of the present invention:
One, interface thread is used to manage all non-query manipulation of this memory database.
1 by the Data Update of data manipulation buffer zone to real time data.
Real time data sequence is outputted to the data encasement block of data query by 2.
3 management data query blocks, data encasement block.Switching is between the two realized after data ready.
Two, query interface only visit data query block.
Query manipulation can carry out at any time, and without any need for mutual exclusion action.
Specific embodiment: a kind of memory database querying method based on key-value pair comprises:
Step 1: with random value by after Hash algorithm for encryption, query key will be formed, and query key and described character string structure are formed a key-value pair;
Step 2: memory database maintenance module arranges the data manipulation buffer zone that memory database maintenance module application memory size can be arranged; Construct a RB_Tree as real time data district; Apply for that one piece of shared drive is as data query district, and data query district head, data query block pointer, data encasement block pointer, data query block, data encasement block are set for data query district;
Step 3: memory database maintenance module opens the Data Update thread for the interface thread and data real-time update carrying out command operating; Described unlatching thread receives data operation commands and data by Socket, and by data manipulation instruction and data write data manipulation buffer zone;
Step 4: client call query interface, query interface, according to the pointed of data query district head, carries out dichotomy inquiry according to key-value pair, and returns Query Result by query interface in data query district.
Wherein, in described step 3, the Data Update thread specific works process of data real-time update is:
Step 31: memory database maintenance module turn-on data more new thread, described Data Update thread, by the data of data manipulation buffer zone, adds real time data district to by data manipulation instruction;
Step 32: after having added data, Data Update thread is by real time data district data sorting and output to data encasement block;
Step 33: after data output to data encasement block completely, Data Update thread exchanges and points to the pointer of data query block and the pointer of data encasement block; The described pointer of sensing data query block and the pointer of data encasement block are arranged in data query build portion (reaching the object exchanging data query block and data encasement block), perform step 31.
The present invention is not limited to aforesaid embodiment.The present invention expands to any new feature of disclosing in this manual or any combination newly, and the step of the arbitrary new method disclosed or process or any combination newly.
Claims (4)
1., based on a memory database querying method for key-value pair, it is characterized in that comprising:
Step 1: with random value by after Hash algorithm for encryption, query key will be formed, and query key and described character string structure are formed a key-value pair;
Step 2: memory database maintenance module arranges the data manipulation buffer zone that memory database maintenance module application memory size can be arranged; Construct a RB_Tree as real time data district; Apply for that one piece of shared drive is as data query district, and data query district head, data query block pointer, data encasement block pointer, data query block, data encasement block are set for data query district;
Step 3: memory database maintenance module opens the Data Update thread for the interface thread and data real-time update carrying out command operating; Described unlatching thread receives data operation commands and data by Socket, and by data manipulation instruction and data write data manipulation buffer zone;
Step 4: client call query interface, query interface, according to the pointed of data query district head, carries out dichotomy inquiry according to key-value pair, and returns Query Result by query interface in data query district.
2. a kind of memory database querying method based on key-value pair according to claim 1, is characterized in that the Data Update thread specific works process of data real-time update in described step 3 is:
Step 31: memory database maintenance module turn-on data more new thread, described Data Update thread, by the data of data manipulation buffer zone, adds real time data district to by data manipulation instruction;
Step 32: after having added data, Data Update thread is by real time data district data sorting and output to data encasement block;
Step 33: after data output to data encasement block completely, Data Update thread exchanges and points to the pointer of data query block and the pointer of data encasement block; The described pointer of sensing data query block and the pointer of data encasement block are arranged in data query build portion (reaching the object exchanging data query block and data encasement block), perform step 31.
3., based on a memory database inquiry unit for key-value pair, it is characterized in that comprising:
Key-value pair forms module, for random value by after Hash algorithm for encryption, forming query key, and query key and described character string structure are formed a key-value pair;
Memory database maintenance module, for arranging the data manipulation buffer zone that memory database maintenance module application memory size can be arranged; Construct a RB_Tree as real time data district; Apply for that one piece of shared drive is as data query district, and data query district head, data query block pointer, data encasement block pointer, data query block, data encasement block are set for data query district; Then memory database maintenance module opens the Data Update thread for the interface thread and data real-time update carrying out command operating; Described unlatching thread receives data operation commands and data by Socket, and by data manipulation instruction and data write data manipulation buffer zone;
Client, for calling query interface, query interface, according to the pointed of data query district head, carries out dichotomy inquiry according to key-value pair, and returns Query Result by query interface in data query district.
4. a kind of memory database inquiry unit based on key-value pair according to claim 1, is characterized in that the Data Update thread specific works process of described data real-time update is:
Step 31: memory database maintenance module turn-on data more new thread, described Data Update thread, by the data of data manipulation buffer zone, adds real time data district to by data manipulation instruction;
Step 32: after having added data, Data Update thread is by real time data district data sorting and output to data encasement block;
Step 33: after data output to data encasement block completely, Data Update thread exchanges and points to the pointer of data query block and the pointer of data encasement block; The described pointer of sensing data query block and the pointer of data encasement block are arranged in data query build portion (reaching the object exchanging data query block and data encasement block), perform step 31.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510907956.4A CN105550271B (en) | 2015-12-10 | 2015-12-10 | A kind of memory database querying method and device based on key-value pair |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510907956.4A CN105550271B (en) | 2015-12-10 | 2015-12-10 | A kind of memory database querying method and device based on key-value pair |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105550271A true CN105550271A (en) | 2016-05-04 |
| CN105550271B CN105550271B (en) | 2018-12-25 |
Family
ID=55829460
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510907956.4A Expired - Fee Related CN105550271B (en) | 2015-12-10 | 2015-12-10 | A kind of memory database querying method and device based on key-value pair |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105550271B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107590199A (en) * | 2017-08-18 | 2018-01-16 | 西安理工大学 | A kind of multi-thread data storehouse design method towards internal memory |
| CN111597163A (en) * | 2020-03-26 | 2020-08-28 | 中国南方航空股份有限公司 | Memory database |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1848111A (en) * | 2005-04-15 | 2006-10-18 | 中兴通讯股份有限公司 | Realizing method of a data operating interface used in internal-memory database |
| CN101963993A (en) * | 2010-10-21 | 2011-02-02 | 江苏科技大学 | Method for fast searching database sheet table record |
| US20110029570A1 (en) * | 2009-07-31 | 2011-02-03 | Worldwide Objects International Limited | Systems and methods for contextualized caching strategies |
| CN104112024A (en) * | 2014-07-30 | 2014-10-22 | 北京锐安科技有限公司 | Method and device for high-performance query of database |
| CN104484400A (en) * | 2014-12-12 | 2015-04-01 | 北京国双科技有限公司 | Method and device for data query processing |
-
2015
- 2015-12-10 CN CN201510907956.4A patent/CN105550271B/en not_active Expired - Fee Related
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1848111A (en) * | 2005-04-15 | 2006-10-18 | 中兴通讯股份有限公司 | Realizing method of a data operating interface used in internal-memory database |
| US20110029570A1 (en) * | 2009-07-31 | 2011-02-03 | Worldwide Objects International Limited | Systems and methods for contextualized caching strategies |
| CN101963993A (en) * | 2010-10-21 | 2011-02-02 | 江苏科技大学 | Method for fast searching database sheet table record |
| CN104112024A (en) * | 2014-07-30 | 2014-10-22 | 北京锐安科技有限公司 | Method and device for high-performance query of database |
| CN104484400A (en) * | 2014-12-12 | 2015-04-01 | 北京国双科技有限公司 | Method and device for data query processing |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107590199A (en) * | 2017-08-18 | 2018-01-16 | 西安理工大学 | A kind of multi-thread data storehouse design method towards internal memory |
| CN107590199B (en) * | 2017-08-18 | 2019-12-24 | 西安理工大学 | Memory-oriented multithreading database design method |
| CN111597163A (en) * | 2020-03-26 | 2020-08-28 | 中国南方航空股份有限公司 | Memory database |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105550271B (en) | 2018-12-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103761318B (en) | A kind of method and system of relationship type synchronization of data in heterogeneous database | |
| EP3702932A1 (en) | Method, apparatus, device and medium for storing and querying data | |
| US9697243B2 (en) | Method and apparatus for searching node by using tree index | |
| CN105843933B (en) | The index establishing method of distributed memory columnar database | |
| CN106815258B (en) | Data storage method and coordination node | |
| US10977251B1 (en) | Join index bitmap for non-equality query conditions | |
| CN103198108B (en) | A kind of index data update method, retrieval server and system | |
| CN111723073B (en) | Data storage processing method, device, processing system and storage medium | |
| CN102946410A (en) | Method and device for network synchronization | |
| CN103136363A (en) | Inquiry processing method and cluster data base system | |
| US11308060B2 (en) | Method, apparatus, device and storage medium for managing index | |
| CN102946411A (en) | Network synchronization system | |
| CN104572862A (en) | Mass data storage access method and system | |
| CN105550271A (en) | Memory database query method and device based on key value pairs | |
| CN113901135A (en) | Cross-source heterogeneous data intelligent visual analysis display method and equipment | |
| CN104462342B (en) | database snapshot synchronization processing method and device | |
| US20150039558A1 (en) | Database management method, database system and medium | |
| CN109410063B (en) | Transaction processing method and device based on block chain and storage medium | |
| CN112395360B (en) | Data synchronization method, device, apparatus and medium based on non-relational database | |
| CN106055646A (en) | Lock-free operation method of concurrent hash tables | |
| CN111078728A (en) | Cross-database query method and device in database filing mode | |
| CN107169097A (en) | A kind of improved method of Spark Broadcasthashjoin operations | |
| CN105630778A (en) | DB data migration method and system | |
| CN103455573A (en) | Database query optimization method based on logic optimization | |
| CN112148739B (en) | Ciphertext index method and system independent of encryption database |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20200806 Address after: Floor 2, No.16, Gaopeng Avenue, high tech Zone, Chengdu, Sichuan 610000 Patentee after: Chengdu Guangda Hengji Communication Technology Co., Ltd Address before: 610041 building four, building two, Shiyang Industrial Park, five village, Shiyang village, fifty-five village, Chengdu hi tech Zone, Sichuan, Qingyun Patentee before: CHENGDU GUANGDA NEW NETWORK TECHNOLOGY Co.,Ltd. |
|
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20181225 Termination date: 20201210 |