CN113127717A - Key retrieval method and system - Google Patents
Key retrieval method and system Download PDFInfo
- Publication number
- CN113127717A CN113127717A CN202110338108.1A CN202110338108A CN113127717A CN 113127717 A CN113127717 A CN 113127717A CN 202110338108 A CN202110338108 A CN 202110338108A CN 113127717 A CN113127717 A CN 113127717A
- Authority
- CN
- China
- Prior art keywords
- key
- memory
- shared memory
- memory database
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 39
- 230000002085 persistent effect Effects 0.000 claims description 9
- 238000013507 mapping Methods 0.000 claims description 8
- 238000003860 storage Methods 0.000 abstract description 11
- 230000006870 function Effects 0.000 abstract description 7
- 238000012545 processing Methods 0.000 abstract description 7
- 238000013461 design Methods 0.000 abstract description 4
- 230000008569 process Effects 0.000 description 12
- 238000010586 diagram Methods 0.000 description 10
- 238000004590 computer program Methods 0.000 description 7
- 238000005516 engineering process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000007792 addition Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/953—Querying, e.g. by the use of web search engines
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/951—Indexing; Web crawling techniques
Abstract
The invention provides a key retrieval method and a key retrieval system. The method and the system apply the memory database, the shared memory and the HashMap to design a secondary cache structure, the key is durably stored in the memory database, when the client is called for the first time, the memory database is mapped into the shared memory, when the client searches, the key is directly searched in the shared memory through an interface provided by the memory database, and only when the key is not searched in the shared memory, the key is inquired in the memory database, so that the quick search of the key is realized. Meanwhile, the method and the system adopt a plurality of machines to deploy the memory database to form a dual-machine hot standby or cluster mode, thereby improving the bearing capacity and the query function of the key, and the memory database is set to be a main standby or cluster mode, realizing the synchronization of the plurality of machines, realizing the storage and retrieval of the key data in a single device by adopting a shared memory Hashmap mode, and greatly improving the processing speed of key retrieval.
Description
Technical Field
The present invention relates to the field of data storage, data retrieval and cryptography, and more particularly, to a key retrieval method and system.
Background
With the rapid development of the information society and the network technology, the internet of things technology is widely and deeply applied to various links of transmission, distribution, use, scheduling and the like of a power system, which means that the number and types of terminal devices accessed to the power network will be greatly increased in the future, massive terminal access brings storage of massive session keys, and how to realize rapid retrieval of the keys is a difficult problem. In the prior art, a technical means for quickly searching a small number of keys is still lacked.
Disclosure of Invention
In order to solve the problem that a rapid retrieval technical means is lacked for massive keys in the prior art, the invention provides a key retrieval method, which comprises the following steps:
step 1, a client accesses a shared memory and retrieves a key from the shared memory, wherein the shared memory is created by a memory database;
step 2, when the secret key does not exist in the shared memory, the client searches in the memory database;
and 3, when the key is retrieved from the memory database, the client synchronizes the key in the shared memory and returns the key.
Further, the client accesses the shared memory by calling a data access interface Key-API, wherein the Key-API is provided by the memory database.
Further, before the client accesses the shared memory, the method further includes:
the method comprises the steps of storing a Key in a memory database of a shared memory server in a persistent mode, automatically loading the Key when the server is powered on, generating a file according to the Key, and mapping the file to the shared memory, wherein the Key is stored in the memory database in a Key-Value mode, a Key is designed as a Key index, a Value is Key data, the Key data is data encrypted by a main Key, and the Key data is inquired according to the index during inquiry and is decrypted by the main Key when the Value is obtained.
Further, when the key stored in the in-memory database is changed, the key-value pairs in the in-memory database and the shared memory are modified simultaneously.
Furthermore, a memory database is deployed by multiple machines to form a dual-machine hot standby or cluster mode.
According to another aspect of the present invention, there is provided a key retrieval system, the system comprising:
the client is used for accessing the shared memory and retrieving the key in the shared memory; when the secret key does not exist in the shared memory, the client side searches in the memory database; when the Key is retrieved from the memory database, synchronizing the Key in a shared memory and returning the Key, wherein the Key-API is provided by the memory database, and the shared memory is created by the memory database;
the server is used for installing a memory database, storing a Key in the memory database in a persistent mode, automatically loading the Key when the server is powered on, generating a file according to the Key, and mapping the file to a shared memory, wherein the Key is stored in the memory database in a Key-Value mode, a Key is designed as a Key index, a Value is Key data, the Key data is data encrypted by a main Key, and the Key data is queried according to the index during query and then decrypted by the main Key when the Value is obtained.
Further, client access to the shared memory is achieved by calling a data access interface, Key-API, provided by the memory database.
Further, when the key stored in the memory database of the server is changed, the key value pairs in the memory database and the shared memory are modified simultaneously.
Furthermore, the server adopts a multi-machine deployment memory database to form a dual-machine hot standby or cluster mode.
Further, the memory database of the server is Redis.
The key retrieval method and the system based on connection provided by the technical scheme of the invention apply the memory database, the shared memory and the HashMap to design a secondary cache structure, the key is durably stored in the memory database, when the client calls for the first time, the memory database is mapped into the shared memory, when the client retrieves, the key is directly retrieved in the shared memory through an interface provided by the memory database, and the key is inquired in the memory database only when the key is not retrieved in the shared memory, thereby realizing the rapid retrieval of the key. Meanwhile, the method and the system adopt a plurality of machines to deploy the memory database to form a dual-machine hot standby or cluster mode, thereby improving the bearing capacity and the query function of the key, and the memory database is set to be a main standby or cluster mode, realizing the synchronization of the plurality of machines, realizing the storage and retrieval of the key data in a single device by adopting a shared memory Hashmap mode, and greatly improving the processing speed of key retrieval.
Drawings
A more complete understanding of exemplary embodiments of the present invention may be had by reference to the following drawings in which:
FIG. 1 is a flow chart of a key retrieval method according to a preferred embodiment of the present invention;
FIG. 2 is a schematic diagram of a key retrieval system according to a preferred embodiment of the present invention;
Detailed Description
The exemplary embodiments of the present invention will now be described with reference to the accompanying drawings, however, the present invention may be embodied in many different forms and is not limited to the embodiments described herein, which are provided for complete and complete disclosure of the present invention and to fully convey the scope of the present invention to those skilled in the art. The terminology used in the exemplary embodiments illustrated in the accompanying drawings is not intended to be limiting of the invention. In the drawings, the same units/elements are denoted by the same reference numerals.
Unless otherwise defined, terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Further, it will be understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense.
Fig. 1 is a flowchart of a key retrieval method according to a preferred embodiment of the present invention. As shown in fig. 1, the key retrieval method according to the preferred embodiment starts at step 101.
In step 101, a client accesses a shared memory and retrieves a key from the shared memory, wherein the shared memory is created by a memory database. And the client side searches the key in the shared memory by adopting a HashMap mode. And when the client retrieves the key in the shared memory, the retrieval is finished.
The memory database ensures parallel operation by multiple processes, which requires the multiple processes to share data in the memory, and the database data is usually huge, so the shared memory has good application value in the memory database. The data of the memory database still needs to be stored on the hard disk, the data used by the examples are all stored in a file, and the file format is determined in the program. When the database is started, a single file will be loaded into the memory. And after the loading is finished, starting the background read-write process. After each process is started, the process address is mapped to the data address of the memory database by using a shared memory mapping function. Thus, instead of having a copy of the data for each process, the processes would operate on a block of memory at the same time. As for the synchronization problem of reading and writing, a bitmap is set in the data file to monitor the available information of the data in the memory. When the process needs to read and write the memory data, the bitmap is checked to find out the information about whether the data is available or not, and then the actual reading is carried out. The synchronization of the bitmap information is controlled by the semaphore.
Hash transforms an input character string of indefinite length into an output character string of definite length through a Hash algorithm, and the character string is a Hash value. This transformation is a compression mapping and the amount of space required for the hash value is typically much smaller than the size of the input data. The HashMap mode adopted in the preferred embodiment is characterized in that:
1) the bottom layer of the Hashmap is stored in an array mode. The key-value key value pair is stored as an element of an array. And loading data from the database through the shared memory server, and mapping the shared memory in the process memory space.
2) key-value is an attribute in map. The key values are stored after being hashed, namely, each key is used for calculating the hash value and then storing the hash value. Each hash value corresponds to an array index, and the array index is calculated according to the hash value and the array length.
3) Since different key values may have the same hash value, i.e., two identical elements appear at a certain position of an array, for this case, the Hashmap is stored in the form of a linked list. Each member in the array is a linked list. All elements accommodated by the data structure contain a pointer for linking between elements. According to the characteristics of the elements, the elements are distributed to different linked lists, and conversely, the correct linked list is found by using a hash algorithm through the characteristics, and then the correct elements are found from the linked list.
The storage structure meets the requirement of fast data query (a data structure which is directly accessed according to the key value), does not occupy too much memory space, and is very convenient.
In step 102, when the key does not exist in the shared memory, the client retrieves from the memory database.
In step 103, when the key is retrieved from the memory database, the client synchronizes the key to the shared memory and returns. And after the client synchronizes the key searched in the memory database with the shared memory, the search is finished.
Preferably, the client accesses the shared memory by calling a data access interface Key-API, wherein the Key-API is provided by the memory database.
The memory database software directly provides a data access interface API for calling the client. The client can perform high-speed examination, addition, deletion, modification and the like. The main tasks of the in-memory database management process include: the method comprises the steps of establishing a shared memory, initializing a memory database, acquiring initial data of a background disk database, monitoring the change of a data table corresponding to the background database, synchronously checking the data of the disk database and the data of the memory database and the like.
Preferably, before the client accesses the shared memory, the method further includes:
the method comprises the steps of storing a Key in a memory database of a shared memory server in a persistent mode, automatically loading the Key when the server is powered on, generating a file according to the Key, and mapping the file to the shared memory, wherein the Key is stored in the memory database in a Key-Value mode, a Key is designed as a Key index, a Value is Key data, the Key data is data encrypted by a main Key, and the Key data is inquired according to the index during inquiry and is decrypted by the main Key when the Value is obtained.
Preferably, the key-value pairs in the in-memory database and the shared memory are modified simultaneously when the key stored in the in-memory database is changed. When the key is changed, the key value pairs in the persistent memory database and the shared memory HashMap can be modified simultaneously, so that the consistency of data is effectively ensured.
Preferably, a memory database is deployed by multiple computers to form a dual-computer hot standby or cluster mode. A multi-machine deployment memory database is adopted to form a dual-machine hot standby mode (two) or a cluster mode (multiple) so as to improve the bearing capacity and the query performance of the secret key. And the memory database is set to be in a main-standby or cluster mode, so that multi-machine synchronization is realized, and retrieval key data is stored in a single device in a shared memory Hashmap mode.
Fig. 2 is a schematic structural diagram of a key retrieval system according to a preferred embodiment of the present invention, as shown in fig. 2, a key retrieval system 200 according to the preferred embodiment includes:
a client 201, configured to access a shared memory 222 and retrieve a key from the shared memory; when the key does not exist in the shared memory, the client searches in the memory database 223; when the key is retrieved from the memory database, synchronizing the key in the shared memory 222 and returning, wherein the shared memory 222 is created by the memory database 223;
the server 202 is configured to install the memory database 223, persistently store the Key in the memory database 223, automatically load the Key when the power is turned on, generate a file according to the Key, and map the file to the shared memory 222, where the Key is stored in the memory database 223 in a Key-Value form, a Key is designed as a Key index, a Value is designed as Key data, the Key data is data encrypted by using a master Key, and the Key data is queried according to the index when the Value is obtained and then decrypted by using the master Key.
The server of the key retrieval system of the preferred embodiment adopts a two-level cache structure design:
1) and storing the data in a memory database in a persistent mode, and automatically loading the data when the data is powered on.
2) The service system is loaded in the shared memory HashMap when being called for the first time, and the query is convenient to accelerate.
3) And modifying the key value pairs in the database of the persistent memory and the HashMap of the shared memory simultaneously when the key is changed so as to ensure the data consistency.
4) And when the key is inquired, the shared memory HashMap is firstly searched, if the key does not exist, the database of the persistent storage memory is continuously inquired, and if the key is inquired, the shared memory HashMap is updated and returned.
The second-level cache structure design adopts a shared memory communication mechanism, data is loaded from a database through a shared memory server, the data is led into a text file, and then the text file is read (mapped) into a shared memory, namely, a tree is built in the memory. The shared memory is mapped in the process memory space, so that the shared memory can be accessed in a common memory access mode, and the processing speed of key retrieval is greatly improved.
Preferably, the client 201 accesses the shared memory by calling a data access interface, Key-API, provided by the memory database. Preferably, when the key stored in the in-memory database of the server 202 is changed, the key-value pairs in the in-memory database and the shared memory are modified at the same time.
Preferably, the server side adopts a multi-machine deployment memory database to form a dual-machine hot standby or cluster mode.
Preferably, the memory database of the server is Redis.
The main points of the main points are that the data of the conventional database is mainly stored on a hard disk, and the data of the main point is mainly stored in the internal memory. Because the IO read-write speed of the hard disk is much slower than that of the CPU and the memory, the main research point of the conventional database in the read-write aspect is to reduce the read-write times of the hard disk as much as possible, and the main research point of the memory database is in the aspects of ensuring real-time data access such as fast algorithm, parallel operation, and the like. Redis is a classic memory database, is an open-source and high-performance key value-based cache and storage system, adapts to cache and storage requirements under different scenes by providing multiple data types, and has obvious advantages in performance compared with other databases based on hard disk storage.
The invention has been described with reference to a few embodiments. However, other embodiments of the invention than the one disclosed above are equally possible within the scope of the invention, as would be apparent to a person skilled in the art from the appended patent claims.
Generally, all terms used in the claims are to be interpreted according to their ordinary meaning in the technical field, unless explicitly defined otherwise herein. All references to "a/an/the [ device, component, etc ]" are to be interpreted openly as referring to at least one instance of said device, component, etc., unless explicitly stated otherwise. The steps of any method disclosed herein do not have to be performed in the exact order disclosed, unless explicitly stated.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solutions of the present invention and not for limiting the same, and although the present invention is described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: modifications and equivalents may be made to the embodiments of the invention without departing from the spirit and scope of the invention, which is to be covered by the claims.
Claims (10)
1. A method for key retrieval, the method comprising:
step 1, a client accesses a shared memory and retrieves a key from the shared memory, wherein the shared memory is created by a memory database;
step 2, when the secret key does not exist in the shared memory, the client searches in the memory database;
and 3, when the key is retrieved from the memory database, the client synchronizes the key in the shared memory and returns the key.
2. The method of claim 1, wherein the client accesses the shared memory by calling a data access interface, Key-API, provided by a memory database.
3. The method of claim 1, wherein before the client accessing the shared memory, further comprising:
the method comprises the steps of storing a Key in a memory database of a shared memory server in a persistent mode, automatically loading the Key when the server is powered on, generating a file according to the Key, and mapping the file to the shared memory, wherein the Key is stored in the memory database in a Key-Value mode, a Key is designed as a Key index, a Value is Key data, the Key data is data encrypted by a main Key, and the Key data is inquired according to the index during inquiry and is decrypted by the main Key when the Value is obtained.
4. The method of claim 3, wherein the key-value pairs in the in-memory database and the shared memory are modified simultaneously when the key stored in the in-memory database is changed.
5. The method of claim 1, wherein the in-memory database is deployed using multiple computers to form a dual-computer hot-standby or cluster mode.
6. A key retrieval system, the system comprising:
the client is used for accessing the shared memory and retrieving the key in the shared memory; when the secret key does not exist in the shared memory, the client side searches in the memory database; when the key is retrieved from the memory database, synchronizing the key in a shared memory and returning, wherein the shared memory is created by the memory database;
the server is used for installing a memory database, storing a Key in the memory database in a persistent mode, automatically loading the Key when the server is powered on, generating a file according to the Key, and mapping the file to a shared memory, wherein the Key is stored in the memory database in a Key-Value mode, a Key is designed as a Key index, a Value is Key data, the Key data is data encrypted by a main Key, and the Key data is queried according to the index during query and then decrypted by the main Key when the Value is obtained.
7. The system of claim 6, wherein client access to the shared memory is achieved by calling a data access interface, Key-API, provided by a memory database.
8. The system of claim 6, wherein the key-value pairs in the in-memory database and the shared memory are modified simultaneously when the key stored in the in-memory database of the server is changed.
9. The system of claim 6, wherein the server deploys the in-memory database by multiple servers to form a dual-server hot-standby or cluster mode.
10. The system according to claim 6, wherein the in-memory database of the server is Redis.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110338108.1A CN113127717A (en) | 2021-03-30 | 2021-03-30 | Key retrieval method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110338108.1A CN113127717A (en) | 2021-03-30 | 2021-03-30 | Key retrieval method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113127717A true CN113127717A (en) | 2021-07-16 |
Family
ID=76775384
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110338108.1A Pending CN113127717A (en) | 2021-03-30 | 2021-03-30 | Key retrieval method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113127717A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114615025A (en) * | 2022-02-22 | 2022-06-10 | 中国建设银行股份有限公司 | Method and device for synchronously processing secret keys |
-
2021
- 2021-03-30 CN CN202110338108.1A patent/CN113127717A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114615025A (en) * | 2022-02-22 | 2022-06-10 | 中国建设银行股份有限公司 | Method and device for synchronously processing secret keys |
| CN114615025B (en) * | 2022-02-22 | 2024-05-03 | 中国建设银行股份有限公司 | Key synchronization processing method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8219544B2 (en) | Method and a computer program product for indexing files and searching files | |
| US9672235B2 (en) | Method and system for dynamically partitioning very large database indices on write-once tables | |
| CN106776967B (en) | Method and device for storing massive small files in real time based on time sequence aggregation algorithm | |
| US10275489B1 (en) | Binary encoding-based optimizations at datastore accelerators | |
| US9576012B2 (en) | Hierarchical tablespace space management | |
| US8868595B2 (en) | Enhanced control to users to populate a cache in a database system | |
| US11580162B2 (en) | Key value append | |
| EP3401807B1 (en) | Synopsis based advanced partition elimination | |
| CN108021717B (en) | Method for implementing lightweight embedded file system | |
| CN107515879B (en) | Method and electronic equipment for document retrieval | |
| US9928178B1 (en) | Memory-efficient management of computer network resources | |
| US10824612B2 (en) | Key ticketing system with lock-free concurrency and versioning | |
| US10990571B1 (en) | Online reordering of database table columns | |
| US20200349163A1 (en) | Data system configured to transparently cache data of data sources and access the cached data | |
| CN106155934A (en) | Based on the caching method repeating data under a kind of cloud environment | |
| CN113377868A (en) | Offline storage system based on distributed KV database | |
| CN112749198A (en) | Multi-level data caching method and device based on version number | |
| US10515055B2 (en) | Mapping logical identifiers using multiple identifier spaces | |
| US10146833B1 (en) | Write-back techniques at datastore accelerators | |
| US10558636B2 (en) | Index page with latch-free access | |
| US10387384B1 (en) | Method and system for semantic metadata compression in a two-tier storage system using copy-on-write | |
| CN114610680A (en) | Method, device and equipment for managing metadata of distributed file system and storage medium | |
| CN113127717A (en) | Key retrieval method and system | |
| CN116842012A (en) | Method, device, equipment and storage medium for storing Redis cluster in fragments | |
| CN106874457B (en) | Method for improving metadata cluster performance through virtual directory |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |