CN112764692A - Method for improving Cache hit rate of Cache wire - Google Patents
Method for improving Cache hit rate of Cache wire Download PDFInfo
- Publication number
- CN112764692A CN112764692A CN202110188532.2A CN202110188532A CN112764692A CN 112764692 A CN112764692 A CN 112764692A CN 202110188532 A CN202110188532 A CN 202110188532A CN 112764692 A CN112764692 A CN 112764692A
- Authority
- CN
- China
- Prior art keywords
- cache
- pool
- data
- improving
- hit rate
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 32
- 241000283973 Oryctolagus cuniculus Species 0.000 claims abstract description 7
- 230000007246 mechanism Effects 0.000 claims abstract description 6
- 230000005012 migration Effects 0.000 claims description 6
- 238000013508 migration Methods 0.000 claims description 6
- 239000012634 fragment Substances 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 230000003068 static effect Effects 0.000 description 4
- 230000006870 function Effects 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000003825 pressing Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0644—Management of space entities, e.g. partitions, extents, pools
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method for improving Cache hit rate of Cache memory, and belongs to the technical field of object storage and Cache. The method for improving Cache Tire Cache hit rate initializes the preheating Cache pool, records the operation prediction IO of the user, and informs the Object Storage Device of the prediction result through Rabbit Mq, so that the Cache pool can advance the Object. The method for improving Cache wire Cache hit rate can increase the whole concurrency of read-write records and improve the efficiency of an event notification mechanism, thereby improving user experience, reducing storage cost and having good popularization and application values.
Description
Technical Field
The invention relates to the technical field of object storage and Cache, and particularly provides a method for improving Cache hit rate of Cache.
Background
The cache is often used to solve the contradiction between the storage performance and the storage cost, and meet the requirement of the user on the storage capacity while ensuring that sufficient IO bandwidth is provided. The Ceph-based storage system also utilizes caching techniques at multiple levels, such as Rados Gateway using distributed caches to store object metadata; adding a cache pool to a data pool by the Object Storage Device to store hot data; the underlying storage adds bcache accelerated read data to the data disk. However, the Cache wire of the Object Storage Device cannot guarantee a high Cache hit rate at present, so that the performance of the Cache Device cannot be fully utilized, and the Cache performance is poor. On one hand, the buffer pool is empty after the buffer pool is initialized, and a few number of IO of the same object cannot promote the object to the buffer pool, so that the buffer pool needs to enter an optimal state after being initialized for a long time, and the IO delay is increased in the period. On the other hand, Cache wire mainly caches recent high-frequency data, and for random and uniform IO, the performance is even worse than that of no Cache, so that the performance needs to be further improved.
Disclosure of Invention
The technical task of the invention is to provide a method for improving Cache hit rate, which can increase the whole concurrency of read-write records and improve the efficiency of an event notification mechanism, thereby improving user experience and reducing storage cost.
In order to achieve the purpose, the invention provides the following technical scheme:
a method for improving Cache hit rate of Cache Tire initializes a preheating Cache pool, records operation prediction IO of a user, and informs an Object Storage Device of a prediction result through RabbitMq, so that the Cache pool can advance an Object.
Preferably, the method for improving the Cache hit rate of the Cache core specifically comprises the following steps:
s1, preheating a Cache memory pool;
s2, recording the operation of the user into a log pool by a Rados Gateway;
s3, transmitting the recorded object information to a Rabbit Mq by Rados Gateway;
s4, the Object Storage Device promotes the Object to the cache pool;
s5, Object Storage Device flushes data back to the data pool.
Preferably, in step S1, after the cache pool is initialized, objects in the data pool are polled, and objects with reference counts larger than a pre-set warm-up threshold are directly loaded into the cache pool.
Preferably, a ceph-object store-tool is adopted to perform data migration between different cache pools when hot data are loaded, write locks are applied to the hot data screened from the data pools during migration, corresponding write requests are blocked, and part of objects of the data pools are set to be read only.
Preferably, step S2 includes the following process:
1) generating a temporary download url for the object;
2) opening the authority operation of reading and writing objects for other users or user groups;
3) setting cross-resource sharing operation;
4) the payment objects of the bucket are added for other users or groups of users.
Preferably, in step S3, when the Rados Gateway initializes, an event notification thread is generated, which reads the omap of the Rados object in the log pool and converts the stored operation record into an event that is generated into the message queue in the RabbitMq by using the event notification mechanism.
Preferably, in step S4, the Object Storage Device consumes the corresponding message on the message queue according to the pg information, converts the message into a rados Object id in the data pool, and promotes the Object from the data pool to the buffer pool.
Preferably, in step S5, after a preset lifting time threshold, objects that are lifted from the data pool to the cache pool without a request hit are marked as cold data and will be brushed into the data pool.
According to the method for improving Cache Tire Cache hit rate, the Cache hit rate is improved through a static mode and a dynamic mode, wherein the static mode refers to that when a Cache pool is initialized, partial hot data are screened out according to reference counting information of objects in a data pool, and the Cache pool is preheated through a direct loading mode. The dynamic mode is that when the cache pool normally operates, a radius Gateway fragment records specific operation of a user and is landed in a log pool, map entries on the fragment are polled at the same time to convert the map entries into events to be produced in a Rabbit Mq message queue, Object Storage Device consumes the events on a corresponding message queue by referring to plan Group information, and forcibly promotes the corresponding objects into the cache pool, and if no data is read or written within a specified time, the objects are considered to be flushed back to the data pool.
Compared with the prior art, the method for improving Cache hit rate of the Cache core has the following outstanding advantages: according to the method for improving the Cache hit rate of the Cache Tire, the Cache pool is preheated by loading the object with high reference count into the Cache pool at one time when the Cache pool is initialized, so that the Cache pool can quickly enter the optimal working state. Meanwhile, the appointed operation is recorded, the data read-write request next by the user is predicted, the object is promoted to the cache pool in advance, and the cache hit rate of random read-write IO is promoted. During recording operation, the data is written into different objects in a slicing mode, the characteristics of a plurality of Rados gateways of the distributed storage system are fully utilized, the integral concurrency of reading and writing records of the system is increased, and the efficiency of an event notification mechanism is improved. The recording operation is implemented by writing the operation data into a log pool, so that the record is not easy to lose after the system is powered off or crashed. The Rabbit Mq serving as the message middleware also meets the requirements of high throughput and low time delay of distributed storage, and has good popularization and application values.
Detailed Description
The method for increasing the Cache hit rate of the Cache sure according to the present invention will be described in further detail with reference to the following embodiments.
Examples
The method for improving Cache Tire Cache hit rate improves the Cache hit rate through a static mode and a dynamic mode, wherein the static mode is to screen out part of hot data according to reference counting information of objects in a data pool when the Cache pool is initialized, and preheat the Cache pool through a direct loading mode. The dynamic mode is that when the cache pool normally operates, a radius Gateway fragment records specific operation of a user and is landed in a log pool, map entries on the fragment are polled at the same time to convert the map entries into events to be produced in a RabbitMq message queue, Object Storage Device consumes the events on a corresponding message queue by referring to plan Group information, and forcibly promotes the corresponding objects into the cache pool, and if no data is read or written within a specified time, the objects are considered to be flushed back to the data pool. The cache pool can quickly enter an optimal state through initialization preheating of the cache pool, next IO is predicted through recording of specific operation of a user, and then prediction result information is notified to the Object Storage Device through Rabbit Mq, so that the cache pool can advance the Object, and the cache hit rate of random read-write IO is improved.
Specifically, the method for improving Cache hit rate of the Cache core comprises the following steps:
s1, preheating a Cache memory pool.
After the cache pool is initialized, before the cache pool starts to work, objects in the data pool are polled, the objects with the reference count larger than a preset preheating threshold value are directly loaded into the cache pool (the preheating threshold value is not selected too small, otherwise, real hot data cannot be screened out, the preheating effect cannot be achieved, a higher cache hit rate cannot be obtained for a long time after the data are loaded), a ceph-object-tool is adopted to carry out data migration among different pools when the hot data are loaded, the hot data screened out from the data pool are locked during the migration, corresponding write-in requests are blocked, and at the moment, part of the objects in the data pool are set to be read only.
S2, Rados Gateway records the operation of the user to the log pool.
The method comprises the following steps:
1) generating a temporary download url for the object;
2) opening the authority operation of reading and writing objects for other users or user groups;
3) setting cross-resource sharing operation;
4) the payment objects of the bucket are added for other users or groups of users.
S3, Rados Gateway sends the recorded object information to rabbitmq.
And generating an event notification thread when the Rados Gateway is initialized, reading the omap (not more than a preset threshold number of entries) of the Rados object in the log pool by the thread, performing truncation if the omap entries are excessive at the moment, and starting reading from the last truncated position when reading next time), converting the operation record stored above into an event to be produced into a message queue in the Rabbit Mq by using an event notification mechanism, wherein an AMQP protocol is adopted in the event sending process, and the efficiency is improved by adopting an asynchronous mode. In order to ensure the retrieval efficiency of the omap item, a callback function is registered for the sending action of the event, and the callback function is responsible for popping up the record to the omap and cleaning the omap space after the event is sent successfully.
S4, Object Storage Device promotes the Object to the buffer pool.
And the Object Storage Device consumes corresponding messages on the message queue according to the pg information (the pg and the message queue are in one-to-one correspondence), and converts the messages into rados Object id in the data pool. And the object is forcibly promoted from the data pool to the cache pool, so that the acceleration is provided for the subsequent reading and writing. Setting a priority lower than the normal promotion in the promotion process, preferentially carrying out normal object promotion when resource competition occurs, simultaneously pressing a request for forced promotion into a queue, and sending a signal to awaken the promotion request again after the resource is released. The read-write request of the object is blocked in the lifting process, so that unknown errors are prevented from occurring.
S5, Object Storage Device flushes data back to the data pool.
The forced promoted object may not be followed by a read-write request, so after a preset promotion time threshold (the threshold is not selected too small, otherwise frequent promotion and refresh operations are caused, and the promotion and refresh operations block the IO request of the client, which may affect the user experience, or too large, otherwise the space of the cache pool may be occupied by cold data, which may affect the cache hit rate while wasting the storage space), the forced promoted object that has not been hit during this period is marked as cold data and refreshed into the data pool (the refresh operation blocks the read-write request of the object, preventing unknown errors from occurring). The above-described embodiments are merely preferred embodiments of the present invention, and general changes and substitutions by those skilled in the art within the technical scope of the present invention are included in the protection scope of the present invention.
Claims (8)
1. A method for improving Cache hit rate of Cache core is characterized in that: the method initializes a preheating buffer pool, records operation prediction IO of a user, and informs an Object Storage Device of a prediction result through RabbitMq, so that the buffer pool can advance an Object.
2. The method for improving the Cache hit rate of the Cache wire according to claim 1, wherein: the method specifically comprises the following steps:
s1, preheating a Cache memory pool;
s2, recording the operation of the user into a log pool by a Rados Gateway;
s3, transmitting the recorded object information to a Rabbit Mq by Rados Gateway;
s4, the Object Storage Device promotes the Object to the cache pool;
s5, Object Storage Device flushes data back to the data pool.
3. The method for improving the Cache hit rate of the Cache wire according to claim 2, wherein: in step S1, after the cache pool is initialized, objects in the data pool are polled, and the objects whose reference count is greater than the pre-set warm-up threshold are directly loaded into the cache pool.
4. The method for improving the Cache hit rate of the Cache wire according to claim 3, wherein: and when the heat data is loaded, a ceph-objectstore-tool is adopted to carry out data migration among different cache pools, write locks are added to the screened heat data in the data pools during migration, corresponding write requests are blocked, and part of objects in the data pools are set to be read only.
5. The method for improving the Cache hit rate of the Cache wire according to claim 4, wherein: step S2 includes the following procedures:
1) generating a temporary download url for the object;
2) opening the authority operation of reading and writing objects for other users or user groups;
3) setting cross-resource sharing operation;
4) the payment objects of the bucket are added for other users or groups of users.
6. The method for improving the Cache hit rate of the Cache wire according to claim 5, wherein: in step S3, when the Rados Gateway is initialized, an event notification thread is generated, which reads the omap of the Rados object in the log pool and converts the stored operation record into an event, which is generated into a message queue in the rabbitmq by using the event notification mechanism.
7. The method for improving the Cache hit rate of the Cache wire according to claim 6, wherein: in step S4, the Object Storage Device consumes the corresponding message in the message queue according to the pg information, converts the message into a rados Object id in the data pool, and promotes the Object from the data pool to the buffer pool.
8. The method for improving the Cache hit rate of the Cache wire according to claim 7, wherein: in step S5, after a preset lifting time threshold, objects that are not hit by a request are lifted from the data pool to the cache pool and marked as cold data and are brushed into the data pool.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110188532.2A CN112764692A (en) | 2021-02-19 | 2021-02-19 | Method for improving Cache hit rate of Cache wire |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110188532.2A CN112764692A (en) | 2021-02-19 | 2021-02-19 | Method for improving Cache hit rate of Cache wire |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112764692A true CN112764692A (en) | 2021-05-07 |
Family
ID=75705581
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110188532.2A Pending CN112764692A (en) | 2021-02-19 | 2021-02-19 | Method for improving Cache hit rate of Cache wire |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112764692A (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103020004A (en) * | 2012-12-14 | 2013-04-03 | 杭州华为数字技术有限公司 | Access method and device for cache coherent-non uniform memory access system |
| CN104793901A (en) * | 2015-04-09 | 2015-07-22 | 北京鲸鲨软件科技有限公司 | Storage device and storage method |
| CN106506668A (en) * | 2016-11-23 | 2017-03-15 | 山东浪潮云服务信息科技有限公司 | A kind of method that object storage is realized based on distributed storage |
| CN109002730A (en) * | 2018-07-26 | 2018-12-14 | 郑州云海信息技术有限公司 | A kind of file system directories right management method, device, equipment and storage medium |
| CN109344092A (en) * | 2018-09-11 | 2019-02-15 | 天津易华录信息技术有限公司 | A kind of method and system improving cold storing data reading speed |
| CN110196689A (en) * | 2019-06-03 | 2019-09-03 | 无锡华云数据技术服务有限公司 | Read method, device and the distributed memory system of data |
| CN110399314A (en) * | 2019-06-25 | 2019-11-01 | 苏州浪潮智能科技有限公司 | A kind of CPU, a kind of electronic equipment and a kind of cpu cache control method |
| CN110502493A (en) * | 2019-08-28 | 2019-11-26 | 北京云端智度科技有限公司 | A kind of buffer efficiency method for improving based on log analysis |
| CN111736767A (en) * | 2020-05-29 | 2020-10-02 | 苏州浪潮智能科技有限公司 | Method and equipment for writing cache of distributed object storage system |
-
2021
- 2021-02-19 CN CN202110188532.2A patent/CN112764692A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103020004A (en) * | 2012-12-14 | 2013-04-03 | 杭州华为数字技术有限公司 | Access method and device for cache coherent-non uniform memory access system |
| CN104793901A (en) * | 2015-04-09 | 2015-07-22 | 北京鲸鲨软件科技有限公司 | Storage device and storage method |
| CN106506668A (en) * | 2016-11-23 | 2017-03-15 | 山东浪潮云服务信息科技有限公司 | A kind of method that object storage is realized based on distributed storage |
| CN109002730A (en) * | 2018-07-26 | 2018-12-14 | 郑州云海信息技术有限公司 | A kind of file system directories right management method, device, equipment and storage medium |
| CN109344092A (en) * | 2018-09-11 | 2019-02-15 | 天津易华录信息技术有限公司 | A kind of method and system improving cold storing data reading speed |
| CN110196689A (en) * | 2019-06-03 | 2019-09-03 | 无锡华云数据技术服务有限公司 | Read method, device and the distributed memory system of data |
| CN110399314A (en) * | 2019-06-25 | 2019-11-01 | 苏州浪潮智能科技有限公司 | A kind of CPU, a kind of electronic equipment and a kind of cpu cache control method |
| CN110502493A (en) * | 2019-08-28 | 2019-11-26 | 北京云端智度科技有限公司 | A kind of buffer efficiency method for improving based on log analysis |
| CN111736767A (en) * | 2020-05-29 | 2020-10-02 | 苏州浪潮智能科技有限公司 | Method and equipment for writing cache of distributed object storage system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106528438A (en) | Segmented junk recovery method for solid-state storage device | |
| CN105677236B (en) | A kind of storage device and its method for storing data | |
| JP2012517644A (en) | Storage system using high-speed storage device as cache | |
| JP2006323826A (en) | System for log writing in database management system | |
| WO2021238260A1 (en) | Pre-read data caching method and apparatus, device, and storage medium | |
| US20120110258A1 (en) | Storage device cache | |
| WO2009065309A1 (en) | Method and system for storing the program and system for the program demanding | |
| WO2011131066A1 (en) | Internet of things system and buffer control method thereof | |
| US20070061509A1 (en) | Power management in a distributed file system | |
| US20050138289A1 (en) | Virtual cache for disk cache insertion and eviction policies and recovery from device errors | |
| US10515671B2 (en) | Method and apparatus for reducing memory access latency | |
| CN111190537B (en) | Method and system for managing sequential storage disk in additional writing scene | |
| WO2024207903A1 (en) | Random write instruction processing method, smr hard disk, and computer device | |
| US6256711B1 (en) | Method for purging unused data from a cache memory | |
| CN112000281A (en) | Caching method, system and device for deduplication metadata of storage system | |
| CN117472285B (en) | Intelligent operation acceleration method for solid state disk, computer equipment and storage medium | |
| CN107402819A (en) | The management method and system of a kind of client-cache | |
| CN116827880B (en) | Cache space management method and device | |
| CN112764692A (en) | Method for improving Cache hit rate of Cache wire | |
| CN105138277A (en) | Cache management method for solid-state disc array | |
| JPH11143779A (en) | Paging processing system for virtual storage device | |
| CN110347339B (en) | Controllable continuous writing method, scheduler and storage system for mechanical hard disk | |
| CN112463306B (en) | Method for sharing disk data consistency in virtual machine | |
| CN113093994A (en) | Data processing method and device | |
| CN114089912A (en) | Data processing method and device based on message middleware and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210507 |
|
| RJ01 | Rejection of invention patent application after publication |