CN110909029A - Method and medium for realizing cache based on Nosql - Google Patents
Method and medium for realizing cache based on Nosql Download PDFInfo
- Publication number
- CN110909029A CN110909029A CN201911089696.9A CN201911089696A CN110909029A CN 110909029 A CN110909029 A CN 110909029A CN 201911089696 A CN201911089696 A CN 201911089696A CN 110909029 A CN110909029 A CN 110909029A
- Authority
- CN
- China
- Prior art keywords
- database
- data
- nosql
- nosql database
- request
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 39
- 238000004364 calculation method Methods 0.000 claims description 5
- 238000003491 array Methods 0.000 claims description 3
- 238000004590 computer program Methods 0.000 claims description 3
- 238000004806 packaging method and process Methods 0.000 claims description 3
- 229910002056 binary alloy Inorganic materials 0.000 claims description 2
- 238000012217 deletion Methods 0.000 description 5
- 230000037430 deletion Effects 0.000 description 5
- 230000004044 response Effects 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 3
- 238000012545 processing Methods 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000009530 blood pressure measurement Methods 0.000 description 1
- 230000006837 decompression Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
Abstract
The invention provides a method and a medium for realizing caching based on a Nosql database, wherein the method specifically comprises the following steps: responding to the data acquisition request, analyzing and classifying the request parameters, and calculating Nosql database keys of different types of request parameters; inquiring a Nosql database according to the calculated key value, and if data are inquired, directly returning the data to the requester; if the data is not inquired, inquiring the data from the database, storing the data into a Nosql database, and returning the data to the requester; correspondingly, the information in the database is updated or deleted according to the data updating or deleting request, the cache information in the Nosql database is deleted according to the corresponding key, and the information in the database corresponding to the cache key in the Nosql database and the request parameter is stored in the Nosql database. The invention improves the speed of data request access and the system concurrency.
Description
Technical Field
The invention belongs to a database storage technology, and particularly relates to a method and a medium for realizing caching based on Nosql.
Background
In the process of project development, data needs to be accessed frequently, if the data is not cached, the query result in the relational database needs to be returned every time, if the relational performance is fixed, the concurrent access capability of the data is improved, and caching is increased to be the fastest access mode. The existing cache modes mainly include two types: one is memory caching of the server, the mode stores data to be cached in a data structure such as ConcurrentHashMap, inquires whether the map exists before inquiring every time, acquires and returns the map, inquires and returns the map after the database is inquired if the map does not exist, and puts the result into the map; however, if this method is used in a distributed system, data in the JVM itself can only be processed if the data is changed (updated or deleted). Although the data update and deletion processing of the distributed system cache can be realized through the message notification mechanism provided by the zookeeper at present, the complexity of system design is increased, and meanwhile, the requirement for maintaining the consistency of data is higher, and the maintenance is inconvenient.
The other method is to realize caching based on a third-party Nosql database, store data to be cached in the Nosql database in the mode, query the Nosql database before each query, and directly return the data according to the calculated key value if the Nosql database exists; if the Nosql database does not exist, returning after inquiring the database, and storing the result into the Nosql database; in the method, before the database is upgraded or released, whether the data exists in the Nosql database needs to be inquired, if the data exists, the data in the database is upgraded or deleted, then the data in the Nosql database is deleted, if the deletion of the Nosql data fails, the transaction will roll back, the upgrading or the deletion will fail, and no influence is caused on the data; in the above processing mode, in a high concurrency situation, there is a multi-write operation for storing data in the Nosql database, since no data exists in the Nosql database when a multi-thread performs query, query SQL is executed instantly and inserted into the Nosql database, and since no data exists in the Nosql when a plurality of threads simultaneously acquire data, it is necessary to query in the relational database and insert data into the Nosql database and the relational database, and thus, there is a situation of writing multiple times simultaneously and successively, but the accuracy of data is not affected.
On the other hand, the amount of cacheable data is limited, and the problem how to store more data based on the limited storage space is also a problem to be solved.
Disclosure of Invention
The invention aims to provide a method for realizing caching based on Nosql.
The technical solution for realizing the purpose of the invention is as follows: a method for realizing caching based on Nosql specifically comprises the following steps:
responding to the data acquisition request, analyzing and classifying the request parameters, and calculating Nosql database keys of different types of request parameters;
inquiring a Nosql database according to the calculated key value, and if data are inquired, directly returning the data to the requester; if the data is not inquired, inquiring the data from the database, storing the data into the Nosql database, simultaneously storing the Nosql database key and the corresponding relation of the request parameters into a relational database, and finally returning the data to the requester;
and responding to the data updating or deleting request, updating or deleting the information in the database, and deleting the cache information in the Nosql database and the corresponding relation information of the cache key and the request parameter in the database.
Preferably, the request parameter types obtained by analyzing and classifying the request parameters include: a numeric ID or a numeric ID array or character type data.
Preferably, the method for calculating the Nosql database key of the numerical ID is as follows: and carrying out hash coding on the numerical ID to obtain a corresponding hash value.
Preferably, the method for calculating the Nosql database key of the numerical ID array is as follows: and carrying out hash coding on the sorted numerical ID to obtain a corresponding hash value.
Preferably, the logarithmic ID implements a self-increasing ordering based on the java Arrays sort.
Preferably, the method for calculating the Nosql database key of the character-type data is as follows: and carrying out hash coding on the sorted character type data to obtain a corresponding hash value.
Preferably, the character-type data is sorted according to rules using the toString method of Java object.
Preferably, the specific method of querying data from the database, storing the data in the Nosql database, and returning the data to the requester is as follows:
inquiring a database according to the request parameters, packaging and compressing the inquiry result, and storing the inquiry result into the Nosql database in a binary system mode;
storing the parameter information and the Nosql database key information into a relational database;
the data is returned to the requestor.
Preferably, the specific method for deleting the cache information in the Nosql database and the cache key information in the database is as follows:
inquiring a corresponding relation table of request parameters and Nosql database keys in a database according to the request parameter information to obtain cache key information corresponding to the request parameters;
and deleting the cache information in the Nosql database according to the inquired cache key information, and deleting the corresponding relation information between the request parameters in the database and the Nosql database key.
Compared with the prior art, the invention has the following remarkable advantages: the invention improves the speed of data request access, improves the system concurrency and the system stability and reduces the dependence degree on the back-end service.
The present invention is described in further detail below with reference to the attached drawings.
Drawings
Fig. 1 is a data acquisition flow chart.
Fig. 2 is a data update or deletion flow diagram.
Detailed Description
As shown in fig. 1 and 2, a method for implementing a cache based on Nosql includes the following specific steps:
responding to a data acquisition request, analyzing and classifying request parameters, and calculating Nosql database keys of different types of request parameters, wherein the specific method comprises the following steps:
analyzing and classifying the request parameters to obtain a numerical ID or a numerical ID array or character data;
calculating the Nosql database key of different types of request parameters, wherein the calculation method comprises the following steps:
numerical ID: and converting the numerical ID into a String ID, and performing hash coding on the String ID to obtain a corresponding hash value.
The Hash value calculating method in String class is to calculate the ASCII value of each character with 31 as weight and to use natural overflow to obtain the equivalent modulus.
Numerical ID array: the numeric IDs or sets of numeric IDs are first sorted, in some embodiments, a self-incrementing sort (single ID without sort) may be implemented based on the java Arrays sort;
the method comprises the steps of calculating the key value of the Nosql database based on the sorted IDs, mainly based on character string Hash codes, md5 and the like, and mainly selecting a unique key value calculation mode based on data scenes, wherein the reason is that the Hash codes are not suitable for uniqueness judgment in some data scenes and are easy to collide, and the keys need to be calculated by md5 or other modes.
Character type parameter: sorting the character type data according to rules by using a toString method of Java object;
and carrying out hash coding or md5 and the like on string returned by the Java object based toString method, and selecting a unique key value calculation mode based on the data scene.
Inquiring a Nosql database according to the calculated key value, and if data are inquired, directly returning the data to the requester; if the data is not inquired, inquiring the data from the relational database, storing the data into the Nosql database, and returning the data to the requester;
in a further embodiment, the specific method for querying data from the relational database, storing the data in the Nosql database, and returning the data to the requester includes:
inquiring a relational database according to the request parameters, and packaging and compressing the inquiry result, wherein a Java comparison universal Snappy decompression mode is selected, the compressed byte array is written into the Nosql database in a binary mode according to bytes, and the data in the Nosql database is read according to the bytes during reading;
storing the request parameter information and the Nosql database key value into a relational database;
returning the data to the requesting party;
updating or deleting the information cached in the relational database and the Nosql database in response to a data update or deletion request;
inquiring corresponding relation data of the request parameters and the Nosql database key values in the relational database according to the request parameter information to obtain the Nosql database key values corresponding to the request parameters;
and deleting cache information in the Nosql database according to the inquired Nosql database key value, and deleting the Nosql database key value in the relational database and the corresponding relational information of the request parameter.
According to the invention, data is cached based on Nosql, so that the data request access speed is increased, the system concurrency and the system stability are improved, and the dependence degree on back-end services is reduced.
The following is to compare response time of data request with that of data request under the condition of closing cache by using the invention, and respectively query the Nosql database in a single-key and multi-key (array) mode, where the comparison is redis and mysql, and the result is shown as a table:
it can be seen from the table that when a request is sent, the data is acquired by using the method, when the number of concurrent users is within 250 and the environment is not changed, all the requests can be returned within 75ms, the request response time and the pressure measurement transaction have no necessary relation, and the essential is to acquire the redis data; and closing the cache, and acquiring data from the mysql, wherein the request response time of the single main key and the multiple main keys is obviously changed, and the response time is obviously increased.
A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, is adapted to carry out the method of sorting goods according to any of the preceding embodiments.
The computer storage media of the present invention may take the form of any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. The computer-readable storage medium may be, for example but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to:
wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
Claims (10)
1. A method for realizing caching based on Nosql is characterized by comprising the following steps:
responding to the data acquisition request, analyzing and classifying the request parameters, and calculating Nosql database keys of different types of request parameters;
inquiring a Nosql database according to the calculated key value, and if data are inquired, directly returning the data to the requester; if the data is not inquired, inquiring the data from the database, storing the data into the Nosql database, simultaneously storing the Nosql database key and the corresponding relation of the request parameters into a relational database, and finally returning the data to the requester;
and responding to the data updating or deleting request, updating or deleting the information in the database, and deleting the cache information in the Nosql database and the corresponding relation information of the cache key and the request parameter in the database.
2. The method of claim 1, wherein the request parameter type obtained by parsing and classifying the request parameter comprises: a numeric ID or a numeric ID array or character type data.
3. The method for realizing caching based on the Nosql database according to claim 2, wherein the calculation method of the Nosql database key of the numerical ID is as follows: and carrying out hash coding on the numerical ID to obtain a corresponding hash value.
4. The Nosql-database-based caching method according to claim 2, wherein the Nosql database key of the numeric ID array is calculated by: and carrying out hash coding on the sorted numerical ID to obtain a corresponding hash value.
5. The method of claim 4, wherein the self-increment ranking of the numeric ID based on java Arrays sort is implemented.
6. The Nosql-database-based caching method according to claim 2, wherein the calculation method of the Nosql database key of the character-type data is: and carrying out hash coding on the sorted character type data to obtain a corresponding hash value.
7. The Nosql database-based caching method according to claim 6, wherein the symbolic data is sorted according to rules by using Javaobject's toString method.
8. The Nosql-database-based caching method according to claim 1, wherein the specific method of querying data from the database, storing the data in the Nosql database, and returning the data to the requestor is:
inquiring a database according to the request parameters, packaging and compressing the inquiry result, and storing the inquiry result into the Nosql database in a binary system mode;
storing the parameter information and the Nosql database key information into a relational database;
the data is returned to the requestor.
9. The Nosql-based caching method according to claim 1, wherein the specific method for deleting the caching information in the Nosql database and the caching key information in the database is:
inquiring a corresponding relation table of request parameters and Nosql database keys in a database according to the request parameter information to obtain cache key information corresponding to the request parameters;
and deleting the cache information in the Nosql database according to the inquired cache key information, and deleting the corresponding relation information between the request parameters in the database and the Nosql database key.
10. A storage medium on which a computer program is stored, which program, when being executed by a processor, is adapted to carry out the method of carrying out caching based on a Nosql database according to any one of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911089696.9A CN110909029A (en) | 2019-11-08 | 2019-11-08 | Method and medium for realizing cache based on Nosql |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911089696.9A CN110909029A (en) | 2019-11-08 | 2019-11-08 | Method and medium for realizing cache based on Nosql |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110909029A true CN110909029A (en) | 2020-03-24 |
Family
ID=69817146
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911089696.9A Pending CN110909029A (en) | 2019-11-08 | 2019-11-08 | Method and medium for realizing cache based on Nosql |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110909029A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113240476A (en) * | 2021-06-15 | 2021-08-10 | 京东数科海益信息科技有限公司 | Method and device for acquiring product valuation data |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103744952A (en) * | 2014-01-02 | 2014-04-23 | 北京集奥聚合网络技术有限公司 | Method and system for implementing multi-table property of NoSQL database |
-
2019
- 2019-11-08 CN CN201911089696.9A patent/CN110909029A/en active Pending
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103744952A (en) * | 2014-01-02 | 2014-04-23 | 北京集奥聚合网络技术有限公司 | Method and system for implementing multi-table property of NoSQL database |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113240476A (en) * | 2021-06-15 | 2021-08-10 | 京东数科海益信息科技有限公司 | Method and device for acquiring product valuation data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10803197B1 (en) | Masking sensitive information in records of filtered accesses to unstructured data | |
| US11176099B2 (en) | Lockless synchronization of LSM tree metadata in a distributed system | |
| US8959075B2 (en) | Systems for storing data streams in a distributed environment | |
| KR101708261B1 (en) | Managing storage of individually accessible data units | |
| US9298775B2 (en) | Changing the compression level of query plans | |
| US8868595B2 (en) | Enhanced control to users to populate a cache in a database system | |
| US20020178341A1 (en) | System and method for indexing and retriving cached objects | |
| US8924373B2 (en) | Query plans with parameter markers in place of object identifiers | |
| US9940360B2 (en) | Streaming optimized data processing | |
| WO2015172533A1 (en) | Database query method and server | |
| US11789923B2 (en) | Compression units in an index block | |
| CN108664596A (en) | The hardware accelerated for table scan | |
| US20200183906A1 (en) | Using an lsm tree file structure for the on-disk format of an object storage platform | |
| US10275486B2 (en) | Multi-system segmented search processing | |
| CN109815240B (en) | Method, apparatus, device and storage medium for managing index | |
| US10664508B1 (en) | Server-side filtering of unstructured data items at object storage services | |
| JP2021002830A (en) | Advanced database decompression | |
| US9646053B2 (en) | OLTP compression of wide tables | |
| US10747773B2 (en) | Database management system, computer, and database management method | |
| US10437806B2 (en) | Database management method and information processing apparatus | |
| CN110909029A (en) | Method and medium for realizing cache based on Nosql | |
| US10366067B2 (en) | Adaptive index leaf block compression | |
| US11928093B2 (en) | Object data stored out of line vector engine | |
| US20160210237A1 (en) | Storage device, data access method, and program recording medium | |
| KR102280443B1 (en) | Cloud database system with multi-cash for reducing network cost in processing select query |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200324 |