CN104572973A - High-performance memory caching system and method - Google Patents
High-performance memory caching system and method Download PDFInfo
- Publication number
- CN104572973A CN104572973A CN201410854596.1A CN201410854596A CN104572973A CN 104572973 A CN104572973 A CN 104572973A CN 201410854596 A CN201410854596 A CN 201410854596A CN 104572973 A CN104572973 A CN 104572973A
- Authority
- CN
- China
- Prior art keywords
- data
- module
- memory
- internal memory
- key
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
- G06F3/0611—Improving I/O performance in relation to response time
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
Abstract
The invention discloses a high-performance memory caching system and method. The high-performance memory caching system and method are achieved based on a data loading module, a data conversion module, a data checkout module, a data storage module and a data retrieval module, convert disk data into KEY-VALUE format and store data in a memory, data redundancy is reduced, the hit rate of retrieval data is improved, the disk access times of an application system are decreased, I/O operation times are decreased, the memory data reading speed is improved by several magnitude orders compared with disk data reading, the service response time of the application system is shortened, and accordingly the performance of the application system is improved.
Description
Technical field
The present invention relates to the memory technology of data, be specifically related to the caching technology of data.
Background technology
Along with the develop rapidly of computer technology, informationalized tide have swepts the globe, get more and more as the self-built application system of enterprises and institutions, as relating to the application system reading data from database or file, can because of business demand, from database or file, read data can increase I/O operation, the speed reading data is low; Special in height is concurrent:
For database, under up to a hundred concurrent connections, can because connect not enough, Database Systems respond slow reason, cause application system to be run slowly, even occur application server collapse phenomenon;
For file, because file reading data, a large amount of I/O can be produced and operate, cause application system to be run slowly, even occur application server collapse phenomenon.
In sum, the efficiency how improving application system reading data is this area urgently technical issues that need to address.
Summary of the invention
For above-mentioned existing application system because reading data from disk, read data rate is caused to reduce, the slow even application server crash issue of system cloud gray model, the object of the present invention is to provide a kind of high performance memory cache mechanism that greatly can improve application system reading data efficiency.
In order to realize above-mentioned mechanism, the invention provides the following two kinds technical scheme:
Scheme 1: a kind of high performance memory cache system, this memory cache system comprises:
Data import module, data in magnetic disk reads in internal memory by described Data import module;
Data conversion module, the data be read in internal memory are converted to the data of KEY-VALUE form by described data conversion module;
Data check module, described data check module carries out uniqueness verification to being converted to KEY-VALUE formatted data;
Data memory module, described data memory module will be converted to KEY-VALUE form and is stored in internal memory through the data that data check module verified;
Data retrieval module; Described data retrieval module is provided for application system retrieving the data retrieval mode storing data in internal memory.
In the preferred version of this system, described data in magnetic disk comprises file, database.
Further, described data check module to be converted to KEY-VALUE formatted data carry out uniqueness verification time, if there are these data in internal memory, then abandon this data, no longer preserve these data in internal memory.
Scheme 2: provide the one that the basis of system is implemented high performance memory cache method in scheme 1, the method comprises the steps:
(1) Data import, is read into data in magnetic disk in internal memory;
(2) data conversion, will be loaded into the data in magnetic disk in internal memory, be converted to the data of KEY-VALUE form;
(3) data check, carries out uniqueness verification to the KEY-VALUE formatted data after conversion in step (2), if there are these data in internal memory, then abandons, no longer repeated to preserve these data in internal memory; Otherwise continue (4);
(4) data store, and are stored in internal memory by the KEY-VALUE formatted data after step (3) inspection;
(5) data retrieval, the data retrieval function of application system invoke memory caching system, retrieves the KEY-VALUE formatted data stored in internal memory and calls.
In the preferred version of the method, described step (1) when application system starts or first access memory data time perform Data import, and only to perform once.
Further, retrieved by major key in described step (5).
Further, conditional information retrieval is carried out in described step (5).
Comprehensively above-mentioned, the present invention has a little following:
1., during digital independent, connect without the need to creating.
2., during digital independent, operate without the need to producing I/O with disk.
3. internal memory is the storage space of CPU energy directly address, and reading rate is fast.
4. the data in internal memory store with KEY-VALUE form, have that inquiry velocity is fast, store data amount large, it is high concurrent to support, is applicable to very much being inquired about by major key, and supports conditions is retrieved.
Accompanying drawing explanation
The present invention is further illustrated below in conjunction with the drawings and specific embodiments.
Fig. 1 is the theory diagram of high performance memory cache system in the present invention;
Fig. 2 is the realization flow figure of high performance memory cache method in the present invention.
Embodiment
The technological means realized to make the present invention, creation characteristic, reaching object and effect is easy to understand, below in conjunction with concrete diagram, setting forth the present invention further.
High performance memory cache mechanism provided by the invention, based on the reading relative to data in magnetic disk, the reading speed of internal storage data will exceed the principle of several order of magnitude, is loaded in internal memory by data in magnetic disk; Data simultaneously in internal memory store according to KEY-VALUE form, decrease data redundancy, improve the hit rate of retrieve data, decrease the number of times of application system access disk, reduce I/O number of operations, then shorten the application system service response time, improve the performance of application system.
Based on above-mentioned principle, high performance memory cache system 10 provided by the invention, primarily of Data import module 11, data conversion module 12, data check module 13, data memory module 14, data retrieval module 15 has cooperatively interacted (as shown in Figure 1).
Wherein, Data import module 11, it is for reading in data in magnetic disk in internal memory.This module when application system starts or first access memory data time perform, and only to perform once.Wherein data in magnetic disk comprises the data in preservation data hereof, database, but is not limited only to the data in file, the data of database, also other data in magnetic disk can be read in internal memory as required.
Data conversion module 12, is converted to the data meeting certain format by the data in magnetic disk be read in internal memory by Data import module 11, as KEY-VALUE formatted data.
Data check module 13, carries out uniqueness verification to the data converting KEY-VALUE form to by data conversion module 12 in internal memory, if there are these data in internal memory, then abandons this data, no longer preserve these data in internal memory, reduces the redundancy of data in EMS memory.
Data memory module 14, by the data being converted to KEY-VALUE form verified through data check module 13, is stored in internal memory with KEY-VALUE form.
Data retrieval module 15, for retrieving the data stored in internal memory, this module faces to application oriented system, and is called by it, facilitates application system to retrieve the data stored in internal memory and call.
Because the data in internal memory store with KEY-VALUE form, application system calling data retrieval module 15 directly carries out the data retrieval of KEY-VALUE form, has the features such as inquiry velocity is fast, store data amount is large, support height concurrent.For this reason, this module support is retrieved by major key, this module also supports conditions retrieval simultaneously.
Below by way of an application example, realize illustrating that the high performance memory cache system of such scheme formation is to realize the process that application system reads at a high speed data.
When application system utilizes above-mentioned high performance memory cache system to read data at a high speed, first high performance memory cache system is installed in corresponding application system.Thus, this application system reads the process following (see Fig. 2) of data:
Step 101, when application system starts, data in magnetic disk reads in internal memory by Data import module.In application system, this step also can be application system first access memory Data Data time perform, and only need perform once, wherein data in magnetic disk comprises the data in preservation data hereof, database.
Step 102, after completing digital independent, the data read in internal memory are changed by data conversion module, all data in magnetic disk read in are converted to the KYE-VALUE data layout in internal memory.
Step 103, after the conversion completing data layout, data check module does certain uniqueness verification to the KEY-VALUE data after conversion, if there are these data in internal memory, then abandon, no longer continued to be saved in internal memory, the redundancy of data in little memory, otherwise continue step 104.
Step 104, by data memory module by step (3) through the KEY-VALUE data that data check module check passes through, be saved in internal memory with KEY-VALUE form, completed the structure of internal storage data thus.
Step 105, if when application system needs access corresponding data, calling data retrieval module, with major key retrieval or the mode of conditional information retrieval, retrieves the KEY-VALUE formatted data stored in internal memory, completes data high-speed and reads.
More than show and describe ultimate principle of the present invention, principal character and advantage of the present invention.The technician of the industry should understand; the present invention is not restricted to the described embodiments; what describe in above-described embodiment and instructions just illustrates principle of the present invention; without departing from the spirit and scope of the present invention; the present invention also has various changes and modifications, and these changes and improvements all fall in the claimed scope of the invention.Application claims protection domain is defined by appending claims and equivalent thereof.
Claims (7)
1. a high performance memory cache system, is characterized in that, described memory cache system comprises:
Data import module, data in magnetic disk reads in internal memory by described Data import module;
Data conversion module, the data be read in internal memory are converted to the data of KEY-VALUE form by described data conversion module;
Data check module, described data check module carries out uniqueness verification to being converted to KEY-VALUE formatted data;
Data memory module, described data memory module will be converted to KEY-VALUE form and is stored in internal memory through the data that data check module verified;
Data retrieval module; Described data retrieval module is provided for application system retrieving the data retrieval mode storing data in internal memory.
2. the high performance memory cache system of one according to claim 1, it is characterized in that, described data in magnetic disk comprises file, database.
3. the high performance memory cache system of one according to claim 1, it is characterized in that, described data check module to be converted to KEY-VALUE formatted data carry out uniqueness verification time, if there are these data in internal memory, then abandon these data, no longer preserve these data in internal memory.
4. a high performance memory cache method, is characterized in that, described method comprises the steps:
(1) Data import, is read into data in magnetic disk in internal memory;
(2) data conversion, will be loaded into the data in magnetic disk in internal memory, be converted to the data of KEY-VALUE form;
(3) data check, carries out uniqueness verification to the KEY-VALUE formatted data after conversion in step (2), if there are these data in internal memory, then abandons, no longer repeated to preserve these data in internal memory; Otherwise continue (4);
(4) data store, and are stored in internal memory by the KEY-VALUE formatted data after step (3) inspection;
(5) data retrieval, the data retrieval function of application system invoke memory caching system, retrieves the KEY-VALUE formatted data stored in internal memory and calls.
5. the high performance memory cache method of one according to claim 4, is characterized in that, described step (1) when application system starts or first access memory data time perform Data import, and only to perform once.
6. the high performance memory cache method of one according to claim 4, is characterized in that, described step is retrieved by major key in (5).
7. the high performance memory cache method of one according to claim 4, is characterized in that, described step carries out conditional information retrieval in (5).
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410854596.1A CN104572973A (en) | 2014-12-31 | 2014-12-31 | High-performance memory caching system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410854596.1A CN104572973A (en) | 2014-12-31 | 2014-12-31 | High-performance memory caching system and method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN104572973A true CN104572973A (en) | 2015-04-29 |
Family
ID=53089035
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410854596.1A Pending CN104572973A (en) | 2014-12-31 | 2014-12-31 | High-performance memory caching system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104572973A (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102693315A (en) * | 2012-05-29 | 2012-09-26 | 上海家配电子商务有限公司 | Method and device for removing URL (uniform resource locator) duplicate on basis of shared memory mapping |
| CN102708148A (en) * | 2012-03-31 | 2012-10-03 | 深圳祥云信息科技有限公司 | Duplication eliminating method based on multidimensional lattice data spatial model |
| CN102902730A (en) * | 2012-09-10 | 2013-01-30 | 新浪网技术(中国)有限公司 | Method and device for reading data based on data cache |
| CN103092775A (en) * | 2013-01-31 | 2013-05-08 | 武汉大学 | Spatial data double cache method and mechanism based on key value structure |
| CN103226609A (en) * | 2013-05-03 | 2013-07-31 | 福建师范大学 | Searching method for WEB focus searching system |
| CN103488581A (en) * | 2013-09-04 | 2014-01-01 | 用友软件股份有限公司 | Data caching system and data caching method |
| CN103530369A (en) * | 2013-10-14 | 2014-01-22 | 浪潮(北京)电子信息产业有限公司 | De-weight method and system |
| CN103544191A (en) * | 2012-07-17 | 2014-01-29 | 人人游戏网络科技发展（上海）有限公司 | Method and device for reading cache data |
| CN103853766A (en) * | 2012-12-03 | 2014-06-11 | 中国科学院计算技术研究所 | Online processing method and system oriented to streamed data |
-
2014
- 2014-12-31 CN CN201410854596.1A patent/CN104572973A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102708148A (en) * | 2012-03-31 | 2012-10-03 | 深圳祥云信息科技有限公司 | Duplication eliminating method based on multidimensional lattice data spatial model |
| CN102693315A (en) * | 2012-05-29 | 2012-09-26 | 上海家配电子商务有限公司 | Method and device for removing URL (uniform resource locator) duplicate on basis of shared memory mapping |
| CN103544191A (en) * | 2012-07-17 | 2014-01-29 | 人人游戏网络科技发展（上海）有限公司 | Method and device for reading cache data |
| CN102902730A (en) * | 2012-09-10 | 2013-01-30 | 新浪网技术(中国)有限公司 | Method and device for reading data based on data cache |
| CN103853766A (en) * | 2012-12-03 | 2014-06-11 | 中国科学院计算技术研究所 | Online processing method and system oriented to streamed data |
| CN103092775A (en) * | 2013-01-31 | 2013-05-08 | 武汉大学 | Spatial data double cache method and mechanism based on key value structure |
| CN103226609A (en) * | 2013-05-03 | 2013-07-31 | 福建师范大学 | Searching method for WEB focus searching system |
| CN103488581A (en) * | 2013-09-04 | 2014-01-01 | 用友软件股份有限公司 | Data caching system and data caching method |
| CN103530369A (en) * | 2013-10-14 | 2014-01-22 | 浪潮(北京)电子信息产业有限公司 | De-weight method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104714972B (en) | Database divides table foundation and querying method | |
| CN102541927B (en) | Method and device for realizing data cache | |
| CN105718455A (en) | Data query method and apparatus | |
| CN106649828B (en) | Data query method and system | |
| WO2017190580A1 (en) | Method and device for accessing database | |
| US20160110224A1 (en) | Generating job alert | |
| US10275481B2 (en) | Updating of in-memory synopsis metadata for inserts in database table | |
| CN104699723A (en) | Data exchange adapter and system and method for synchronizing data among heterogeneous systems | |
| CN107391544B (en) | Processing method, device and equipment of column type storage data and computer storage medium | |
| CN106909595B (en) | Data migration method and device | |
| CN103729453A (en) | HBase table conjunctive query optimization method | |
| CN103473314A (en) | Key value pair storing method and device based on shared memory | |
| CN103914483A (en) | File storage method and device and file reading method and device | |
| CN102129425A (en) | Method and device for accessing large-object set table in data warehouse | |
| CN107729504A (en) | A kind of method and system for handling large data objectses | |
| CN103793475A (en) | Distributed file system data migration method | |
| CN105573673A (en) | Database based data cache system | |
| CN102467523A (en) | Method for establishing index files and method for querying data blocks by using index files | |
| CN101620634A (en) | Data statistics device and data statistics method | |
| CN105808451A (en) | Data caching method and related apparatus | |
| CN108170376A (en) | The method and system that storage card is read and write | |
| US10871921B2 (en) | Method and system for facilitating atomicity assurance on metadata and data bundled storage | |
| CN107220342A (en) | The control method and system of a kind of distributed data base | |
| CN103870605A (en) | Data base virtual image achieving method orientated to object | |
| CN104572973A (en) | High-performance memory caching system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20150429 |
|
| RJ01 | Rejection of invention patent application after publication |