CN105528181A - Data IO hash method - Google Patents
Data IO hash method Download PDFInfo
- Publication number
- CN105528181A CN105528181A CN201510886346.0A CN201510886346A CN105528181A CN 105528181 A CN105528181 A CN 105528181A CN 201510886346 A CN201510886346 A CN 201510886346A CN 105528181 A CN105528181 A CN 105528181A
- Authority
- CN
- China
- Prior art keywords
- cache
- hash
- data
- blocks
- buffer memory
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
- G06F16/137—Hash-based
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
- G06F3/0656—Data buffering arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
- G06F3/0674—Disk device
Abstract
The present invention discloses a data IO hash method. The method comprises the steps of determining the hash number of an accessed data block in a cache set cache_set by a cache hash algorithm when a data I/O request is issued to a cache; and transferring data from a cache block to a disk block corresponding to the cache block. According to the method, performance influence caused by access conflict of the disk block can be effectively reduced, and cache performance is improved.
Description
Technical field
The present invention relates to stocking system Performance Optimization technique field, particularly relate to a kind of method of data IO hash.
Background technology
At present, along with the arrival of digital times, in daily life and scientific research, increasing traditional business starts digitizing, networking, data explosion formula is impelled to increase, the status of storage system in whole transaction processing system also ever more important, but data stream causes storage system to produce the bottleneck problem of serious I/O input and output endlessly.
The performance of computer system determines primarily of performance two parts of processing subsystem performance and I/O subsystem.Wherein the processing speed of CPU has remained growth at a high speed, and although its memory capacity of I/O subsystem increases very fast, but the growth of its processing speed does not far catch up with the growth of CPU speed, for solving the problem, in modem computer systems, from register, L1/L2 high-speed cache, internal memory, flash memory, to disk/optical disc/storage networking, memory hardware at different levels constitutes a pyramid structure, and bottom memory capacity is larger, and access speed is also slower.
Wherein, the performance acceleration effect of caching system, the performance impact for whole system is obvious, and the key factor affecting caching performance is disk block access conflict and cache hit rate.General, the reading of data IO is introduced into cache blocks, disk is entered from cache blocks, the linear relationship of disk block and cache blocks is set up by simple mathematical operation, although most type of service can be adapted to, but be difficult to adapt to some particular service types, the fast access conflict of disk can be caused in some cases serious, thus affect the performance of whole system.
Summary of the invention
The object of this invention is to provide a kind of method of data IO hash, to realize significantly reducing the performance impact that the fast access conflict of disk causes, promote caching performance.
For solving the problems of the technologies described above, the invention provides a kind of method of data IO hash, the method comprises:
Ask to be issued to buffer memory when there being data I/O after, determine the hash numbering of accessed data block in cache set cache_set by buffer memory hashing algorithm;
Data are migrated to the disk block corresponding with cache blocks from cache blocks.
Preferably, described by buffer memory hashing algorithm determine accessed data block in cache set cache_set hash numbering, comprising:
Utilize buffer memory hashing algorithm to calculate hashed value corresponding to cache blocks linear number, determine the hash numbering of accessed data block in cache set cache_set.
Preferably, before the hashed value that described calculating cache blocks linear number is corresponding, also comprise:
The start sector number dbn to disk operating is extracted by data I/O request.
Preferably, the described data I/O of passing through request also comprises after extracting the start sector number dbn to disk operating:
The linear number of cache set cache_set corresponding to initial sector is calculated by the sum of start sector number dbn and cache set cache_set.
Preferably, the described sum by start sector number dbn and cache set cache_set also comprises after calculating the linear number of cache set cache_set corresponding to initial sector:
Utilize buffer memory hashing algorithm to calculate start sector number dbn cache blocks linear number corresponding in cache set cache_set, determine the hash numbering of the cache set cache_set that initial sector is corresponding.
Preferably, described by data from before cache blocks migrates to the disk block corresponding with cache blocks, also comprise:
Obtain the interface of cache set cache_set parameter configuration.
Preferably, after the interface of described acquisition cache set cache_set parameter configuration, also comprise:
By the number of cache blocks in the interface determination cache set of cache set cache_set parameter configuration;
By the size of cache blocks in the interface adjustment cache set of cache set cache_set parameter configuration.
The method of a kind of data IO hash provided by the present invention, after asking to be issued to buffer memory, determines the hash numbering of accessed data block in cache set cache_set by buffer memory hashing algorithm when there being data I/O; Data are migrated to the disk block corresponding with cache blocks from cache blocks.Visible, by hashing algorithm, the linear corresponding relation between fast to cache blocks and disk is improved, disk block hash corresponding for cache blocks is come, significantly reduce the performance impact that the fast access conflict of disk causes, also remarkable result is had in addition for lifting cache hit rate, so set up the hash corresponding relation between cache blocks and disk block by hashing algorithm, the disk block scope that same buffered is corresponding can be promoted, and there is remarkable result for avoiding cache blocks access conflict, the cache hit rate of whole system can be effectively provided, improve the performance of whole caching system, realization significantly reduces the performance impact that the fast access conflict of disk causes, promote caching performance.
Accompanying drawing explanation
In order to be illustrated more clearly in the embodiment of the present invention or technical scheme of the prior art, be briefly described to the accompanying drawing used required in embodiment or description of the prior art below, apparently, accompanying drawing in the following describes is only embodiments of the invention, for those of ordinary skill in the art, under the prerequisite not paying creative work, other accompanying drawing can also be obtained according to the accompanying drawing provided.
Fig. 1 is the process flow diagram of the method for a kind of data IO hash provided by the present invention;
Fig. 2 is the schematic diagram of cache set hash of the present invention;
Fig. 3 is the schematic diagram of cache blocks hash of the present invention.
Embodiment
Core of the present invention is to provide a kind of method of data IO hash, to realize significantly reducing the performance impact that the fast access conflict of disk causes, promotes caching performance.
The present invention program is understood better in order to make those skilled in the art person, below in conjunction with the accompanying drawing in the embodiment of the present invention, technical scheme in the embodiment of the present invention is clearly and completely described, obviously, described embodiment is only the present invention's part embodiment, instead of whole embodiments.Based on the embodiment in the present invention, those of ordinary skill in the art, not making the every other embodiment obtained under creative work prerequisite, belong to the scope of protection of the invention.
Please refer to Fig. 1, Fig. 1 is the process flow diagram of the method for a kind of data IO hash provided by the present invention, and the method comprises:
S11: ask to be issued to buffer memory when there being data I/O after, determines the hash numbering of accessed data block in cache set cache_set by buffer memory hashing algorithm;
Wherein, determine that the process of the hash numbering of accessed data block in cache set cache_set is specially by buffer memory hashing algorithm: utilize buffer memory hashing algorithm to calculate hashed value corresponding to cache blocks linear number, determine the hash numbering of accessed data block in cache set cache_set.
Before the hashed value utilizing buffer memory hashing algorithm calculating cache blocks linear number corresponding, also comprise: extract the start sector number dbn to disk operating by data I/O request.
In addition, after extracting the start sector number dbn to disk operating by data I/O request, the linear number of cache set cache_set corresponding to initial sector is calculated by the sum of start sector number dbn and cache set cache_set.
After calculating the linear number of cache set cache_set corresponding to initial sector by the sum of start sector number dbn and cache set cache_set, utilize buffer memory hashing algorithm to calculate start sector number dbn cache blocks linear number corresponding in cache set cache_set, determine the hash numbering of the cache set cache_set that initial sector is corresponding.
S12: data are migrated to the disk block corresponding with cache blocks from cache blocks.
Wherein, by data from before cache blocks migrates to the disk block corresponding with cache blocks, obtain the interface of cache set cache_set parameter configuration.
After obtaining the interface of cache set cache_set parameter configuration, by the number of cache blocks in the interface determination cache set of cache set cache_set parameter configuration, by the size of cache blocks in the interface adjustment cache set of cache set cache_set parameter configuration.
Visible, linear corresponding relation between fast to cache blocks and disk is improved by hashing algorithm by the method, disk block hash corresponding for cache blocks is come, significantly reduces the performance impact that the fast access conflict of disk causes, also have remarkable result for lifting cache hit rate in addition.
Concrete, broken by the linear corresponding relation of hashing algorithm by cache blocks and accessed disk block, re-establish the corresponding relation of hash, the disk block scope that same buffered is corresponding can be promoted, and there is remarkable result for avoiding cache blocks access conflict, the cache hit rate of whole system effectively can be provided, improve the performance of whole caching system.Effectively ensure the demand of business from buffer memory aspect, reduce the delay of business, accelerate the response time of business, and effectively can improve the utilization factor of buffer memory.
In more detail, the detailed process of the method is:
(1) after having I/O to ask to be issued to buffer memory, the start sector number dbn to disk operating is extracted by I/O;
(2) linear number of the corresponding cache set cache_set in initial sector is calculated by start sector number dbn and cache set cache_set sum;
(3) by buffer memory hashing algorithm, calculate the hashed value of cache set cache_set linear number, determine the hash numbering of the corresponding cache set cache_set in initial sector;
(4) start sector number dbn cache blocks linear number corresponding in cache set cache_set is calculated further;
(5) by buffer memory hashing algorithm, calculate the hashed value that cache blocks linear number is corresponding, determine the hash numbering of accessed data block in cache set cache_set, data are migrated to the disk block corresponding with cache blocks from cache blocks.
The method of a kind of data IO hash that the present invention proposes, the linear corresponding relation between cache blocks and disk block is broken by buffer memory hashing algorithm, cache blocks is made to become hash state with the corresponding relation of disk block, the disk areas that such design can make same buffered corresponding is more extensive, thus effectively prevent the stand-by period that disk block access conflict causes, effectively can improve caching performance, cache hit rate also be had and promotes effect significantly.
Concrete, in order to realize function of the present invention, method of the present invention can be applicable to cache set molded tissue block, cache blocks molded tissue block, cache set Hash module and cache blocks Hash module.
Wherein, cache set molded tissue block comprises two functions: one is the organization and administration function of cache set cache_set; Two interfaces being to provide cache set cache_set parameter configuration, wherein cache set cache_set comprises multiple cache blocks, and wherein, in cache set, the number of cache blocks can pass through interface configuration.
Wherein, cache blocks molded tissue block has two functions too: the organization and administration function of consciousness cache blocks; Two management interfaces being to provide cache blocks relevant configured parameter, the size of cache blocks can be adjusted by interface.
Wherein, cache set Hash module primary responsibility realizes the hash of cache set, cache set number and initial sector can draw the linear corresponding relation of accessed disk block and cache set after simple mathematical operation, cache set Hash module can break this linear relationship to by hashing algorithm subsequently, re-establishes new hash corresponding relation.
Wherein, the function of cache blocks Hash module is the hash corresponding relation realized between the cache blocks of certain cache set inside and disk block.
Fig. 2 is the schematic diagram of cache set hash of the present invention, and wherein dotted portion is the linear corresponding relation of cache set and disk block, has rebuild the corresponding relation of cache set and disk block by buffer memory hashing algorithm, is no longer simple linear corresponding between the two.
Fig. 3 is cache blocks hash schematic diagram of the present invention, similar with Fig. 2, the cache blocks of cache set inside and the corresponding relation of disk block were originally the linear corresponding relation in figure shown in dotted line, recombinated in this design, by hashing algorithm, its corresponding relation were become unordered hash correspondence.
The idiographic flow carrying out I/O hash after I/O request is issued to buffer memory is: first carry out Hash to cache set, set up the hash corresponding relation of cache set and disk block; Then hashing operation is carried out to each cache blocks in cache set, allow the corresponding relation between each cache blocks with disk become without the need to hash corresponding.
To sum up, the method for a kind of data IO hash provided by the present invention, after asking to be issued to buffer memory, determines the hash numbering of accessed data block in cache set cache_set by buffer memory hashing algorithm when there being data I/O; Data are migrated to the disk block corresponding with cache blocks from cache blocks.Visible, by hashing algorithm, the linear corresponding relation between fast to cache blocks and disk is improved, disk block hash corresponding for cache blocks is come, significantly reduce the performance impact that the fast access conflict of disk causes, also remarkable result is had in addition for lifting cache hit rate, so set up the hash corresponding relation between cache blocks and disk block by hashing algorithm, the disk block scope that same buffered is corresponding can be promoted, and there is remarkable result for avoiding cache blocks access conflict, the cache hit rate of whole system can be effectively provided, improve the performance of whole caching system, realization significantly reduces the performance impact that the fast access conflict of disk causes, promote caching performance.
Above the method for a kind of data IO hash provided by the present invention is described in detail.Apply specific case herein to set forth principle of the present invention and embodiment, the explanation of above embodiment just understands method of the present invention and core concept thereof for helping.It should be pointed out that for those skilled in the art, under the premise without departing from the principles of the invention, can also carry out some improvement and modification to the present invention, these improve and modify and also fall in the protection domain of the claims in the present invention.
Claims (7)
1. a method for data IO hash, is characterized in that, comprising:
Ask to be issued to buffer memory when there being data I/O after, determine the hash numbering of accessed data block in cache set cache_set by buffer memory hashing algorithm;
Data are migrated to the disk block corresponding with cache blocks from cache blocks.
2. the method for claim 1, is characterized in that, described by buffer memory hashing algorithm determine accessed data block in cache set cache_set hash numbering, comprising:
Utilize buffer memory hashing algorithm to calculate hashed value corresponding to cache blocks linear number, determine the hash numbering of accessed data block in cache set cache_set.
3. method as claimed in claim 2, is characterized in that, before the described hashed value utilizing buffer memory hashing algorithm calculating cache blocks linear number corresponding, also comprises:
The start sector number dbn to disk operating is extracted by data I/O request.
4. method as claimed in claim 3, is characterized in that, the described data I/O of passing through request also comprises after extracting the start sector number dbn to disk operating:
The linear number of cache set cache_set corresponding to initial sector is calculated by the sum of start sector number dbn and cache set cache_set.
5. method as claimed in claim 4, is characterized in that, the described sum by start sector number dbn and cache set cache_set also comprises after calculating the linear number of cache set cache_set corresponding to initial sector:
Utilize buffer memory hashing algorithm to calculate start sector number dbn cache blocks linear number corresponding in cache set cache_set, determine the hash numbering of the cache set cache_set that initial sector is corresponding.
6. the method for claim 1, is characterized in that, described by data from before cache blocks migrates to the disk block corresponding with cache blocks, also comprise:
Obtain the interface of cache set cache_set parameter configuration.
7. method as claimed in claim 6, is characterized in that, after the interface of described acquisition cache set cache_set parameter configuration, also comprise:
By the number of cache blocks in the interface determination cache set of cache set cache_set parameter configuration;
By the size of cache blocks in the interface adjustment cache set of cache set cache_set parameter configuration.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510886346.0A CN105528181B (en) | 2015-12-04 | 2015-12-04 | A kind of method of data IO hash |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510886346.0A CN105528181B (en) | 2015-12-04 | 2015-12-04 | A kind of method of data IO hash |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105528181A true CN105528181A (en) | 2016-04-27 |
| CN105528181B CN105528181B (en) | 2019-05-10 |
Family
ID=55770435
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510886346.0A Active CN105528181B (en) | 2015-12-04 | 2015-12-04 | A kind of method of data IO hash |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105528181B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109376097A (en) * | 2018-08-31 | 2019-02-22 | 华东计算技术研究所（中国电子科技集团公司第三十二研究所） | Method for solving hash search learning and address aging conflict of multiple microengines |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7685312B1 (en) * | 2005-02-10 | 2010-03-23 | Sun Microsystems, Inc. | Resource location by address space allocation |
| CN103019963A (en) * | 2012-12-31 | 2013-04-03 | 华为技术有限公司 | Cache mapping method and storage device |
| CN103279561A (en) * | 2013-06-13 | 2013-09-04 | 三珠数码软件开发（上海）有限公司 | Method for increasing random database data read-write speed |
| CN103678166A (en) * | 2013-08-16 | 2014-03-26 | 记忆科技(深圳)有限公司 | Method and system for using solid-state disk as cache of computer |
| CN103729442A (en) * | 2013-12-30 | 2014-04-16 | 华为技术有限公司 | Method for recording event logs and database engine |
-
2015
- 2015-12-04 CN CN201510886346.0A patent/CN105528181B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7685312B1 (en) * | 2005-02-10 | 2010-03-23 | Sun Microsystems, Inc. | Resource location by address space allocation |
| CN103019963A (en) * | 2012-12-31 | 2013-04-03 | 华为技术有限公司 | Cache mapping method and storage device |
| CN103279561A (en) * | 2013-06-13 | 2013-09-04 | 三珠数码软件开发（上海）有限公司 | Method for increasing random database data read-write speed |
| CN103678166A (en) * | 2013-08-16 | 2014-03-26 | 记忆科技(深圳)有限公司 | Method and system for using solid-state disk as cache of computer |
| CN103729442A (en) * | 2013-12-30 | 2014-04-16 | 华为技术有限公司 | Method for recording event logs and database engine |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109376097A (en) * | 2018-08-31 | 2019-02-22 | 华东计算技术研究所（中国电子科技集团公司第三十二研究所） | Method for solving hash search learning and address aging conflict of multiple microengines |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105528181B (en) | 2019-05-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11307769B2 (en) | Data storage method, apparatus and storage medium | |
| CN103856567B (en) | Small file storage method based on Hadoop distributed file system | |
| CN103617131B (en) | Data caching achieving method | |
| CN103885728A (en) | Magnetic disk cache system based on solid-state disk | |
| CN108920387B (en) | Method and device for reducing read delay, computer equipment and storage medium | |
| CN104572505B (en) | System and method for ensuring eventual consistency of mass data caches | |
| WO2017041570A1 (en) | Method and apparatus for writing data to cache | |
| CN103246696A (en) | High-concurrency database access method and method applied to multi-server system | |
| CN103150395B (en) | Directory path analysis method of solid state drive (SSD)-based file system | |
| CN106445405A (en) | Flash storage-oriented data access method and apparatus | |
| Lee et al. | ActiveSort: Efficient external sorting using active SSDs in the MapReduce framework | |
| CN103716375A (en) | Distributed cache-based multistage financial data interaction method and system | |
| CN103559017A (en) | Character string matching method and system based on graphic processing unit (GPU) heterogeneous computing platform | |
| CN106469123A (en) | A kind of write buffer distribution based on NVDIMM, method for releasing and its device | |
| CN101763390A (en) | Database storing system and method based on Berkeley DB | |
| CN105468297A (en) | Quick synchronization method for master and slave device data in cloud storage system | |
| CN103617195B (en) | A kind of data synchronous system and method for MySQL database | |
| CN109408411A (en) | The L1 Cache management method of GPGPU based on data access number | |
| CN106201918B (en) | A kind of method and system based on big data quantity and extensive caching quick release | |
| Papagiannis et al. | Kreon: An efficient memory-mapped key-value store for flash storage | |
| CN106469119A (en) | A kind of data write buffer method based on NVDIMM and its device | |
| CN104598396B (en) | A kind of system and method for ensureing mass data caching real-time and uniformity | |
| CN105528181A (en) | Data IO hash method | |
| CN108958667A (en) | A kind of method for reading data based on distributed memory system, system and device | |
| KR101830504B1 (en) | In-Memory DB Connection Support Type Scheduling Method and System for Real-Time Big Data Analysis in Distributed Computing Environment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |