CN108958667A - A kind of method for reading data based on distributed memory system, system and device - Google Patents
A kind of method for reading data based on distributed memory system, system and device Download PDFInfo
- Publication number
- CN108958667A CN108958667A CN201810929204.1A CN201810929204A CN108958667A CN 108958667 A CN108958667 A CN 108958667A CN 201810929204 A CN201810929204 A CN 201810929204A CN 108958667 A CN108958667 A CN 108958667A
- Authority
- CN
- China
- Prior art keywords
- data
- read operation
- operation request
- pgcache
- reading
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
- G06F3/0656—Data buffering arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention proposes a kind of method for reading data based on distributed memory system, comprising: receives the read operation request that client is sent；Count the temperature that data are read in the read operation request；Judge whether current operation is read operation, if it is not, then continuing to the read operation request of client transmission；If it is, the object data that whether there is the read operation request in pgcache is judged, if it is present to read the content of data described in assembling；If it does not exist, then handling read operation request using OSD, after the completion of processing, the content of data is read described in assembling.It can be seen that, this method on main pg by caching some dsc datas, and carry out data aging and dsc data update in pgcache in time with the reading of data, reading delay can be effectively reduced, the data reading speed of distributed memory system is improved, and then improves the readwrite performance of system entirety.
Description
Technical field
The present invention relates to technical field of data storage, more particularly to a kind of number based on distributed memory system
According to read method, system and device.
Background technique
With the development of computer technology, the data of server storage are more and more, and individual server is just had no idea full
The requirement of sufficient storing data, therefore develop distributed memory system.
In current common distributed memory system, it is contemplated that the reliability of data storage commonly uses correcting and eleting codes technology pair
Data do certain redundancy to increase the reliability of system, are K data block by the data cutting of storage, pass through encryption algorithm meter
M check block is calculated, and is stored in entangle and delete in data pool, can restore original using any K data block in K+M data block
Data.Rule settings are deleted by entangling for the correcting and eleting codes, so that it may be controlled storage size, be conducive to the utilization for improving storage
Rate.
But it since existing processing logic is to create to entangle using fixed stripe size to delete data pool, can not be directed to
Different stripe sizes is arranged in different data, also can not just adapt to different types of file storage, cause space utilization rate
Decline.The decline of space utilization rate can bring the read-write operation of redundant data, and the readwrite performance of system entirety is caused to decline.Cause
This, entangled on distributed memory system delete data pool read operation time delay it is larger, user's access entangles when deleting data pool response too
Slowly, user experience has been seriously affected.
Summary of the invention
In view of the above problems, the purpose of the present invention is to provide a kind of reading data sides based on distributed memory system
Method, system and device, by main logic storage unit PlacementGroup(abbreviation PG) on cache some dsc datas (one
Access times of the object in regular hour section number indicate its temperature), effectively reduce reading delay, sufficiently mention
High user experience.
The present invention to achieve the above object, is achieved through the following technical solutions: a kind of number based on distributed memory system
According to read method, include the following steps:
Step 1: receiving the read operation request that client is sent；
Step 2: counting the temperature that data are read in the read operation request；
Step 3: judging whether current operation is read operation, if so, going to step 4；If it is not, then going to step 1；
Step 4: the object data that whether there is the read operation request in pgcache is judged, if it is present going to step 6；
If it does not exist, then going to step 5；
Step 5: handling read operation request using OSD, step 4 is gone to after the completion of processing；
Step 6: the content of data is read described in assembling；
Step 7: reading and complete, return to client.
Further, described to handle read operation request using OSD, it specifically includes:
Step 51: the object that whether there is read operation request in pgcache is judged, if it is present going to step 54；If no
In the presence of then going to step 52；
Step 52: judging whether pgcache reaches thresholding, if it is, going to step 53；If it is not, then going to step 54；
Step 53: deleting the smallest data of temperature in pgcache；
Step 54: updating the object data of read operation request described in pgcache.
Further, it is described influence data temperature operation, comprising: write, truncate, writefull, zero or
delete。
Further, if be abnormal when handling read operation request using OSD, by file of the pgcache on SSD
It is purged.
In order to solve the above technical problems, the present invention also provides a kind of reading data system based on distributed memory system
System, comprising: request receiving module, data hot statistics module, data judgment module and data assembling module；
The request receiving module is used to receive the read operation request of client transmission；
The data hot statistics module is for counting the temperature that read data in the read operation request；
The data judgment module is used to judge to whether there is in pgcache the object data of the read operation request；
The data assembling module is used to assemble the content that read data.
Further, further includes: object judgment module, threshold decision block, data aging module and data update module,
The object judgment module is used to judge to whether there is in pgcache the object of read operation request；
The threshold decision block is for judging whether pgcache reaches thresholding；
The data aging module is for deleting the smallest data of temperature in pgcache；
The data update module is used to update the object data of read operation request described in pgcache.
Further, further includes: file removes module, will if be abnormal when for handling read operation request using OSD
File of the pgcache on SSD is purged.
In order to solve the above technical problems, the present invention also provides a kind of, the reading data based on distributed memory system is filled
It sets, comprising:
Memory, for storing computer program；
Processor realizes the above-mentioned method for reading data based on distributed memory system for executing the computer program
Step.
The prior art is compared, the beneficial effects of the invention are that: it is proposed by the present invention a kind of based on distributed memory system
Method for reading data, comprising: receive the read operation request that client is sent；Data will be read by counting in the read operation request
Temperature；Judge whether current operation is read operation, if it is not, then continuing to the read operation request of client transmission；If
It is the object data for then judging to whether there is in pgcache the read operation request, if it is present to be read described in assembling
The content of data；If it does not exist, then handling read operation request using OSD, after the completion of processing, data are read described in assembling
Content.It can be seen that this method carries out in time by caching some dsc datas on main pg, and with the reading of data
Data aging and dsc data in pgcache update, and can effectively reduce reading delay, improve the number of distributed memory system
According to reading speed, and then improve the readwrite performance of system entirety.In addition, the present invention also provides be based on distributed memory system
Data reading system and device, beneficial effect is as described above.
It can be seen that compared with prior art, the present invention implementing with substantive distinguishing features outstanding and significant progress
Beneficial effect be also obvious.
Detailed description of the invention
Attached drawing 1 is the method flow diagram of the embodiment of the present invention one.
Attached drawing 2 is the method flow diagram of the embodiment of the present invention two.
Attached drawing 3 is system structure diagram of the invention.
Attached drawing 4 is the device of the invention structural schematic diagram.
Specific embodiment
It explains with reference to the accompanying drawing to a specific embodiment of the invention.
Embodiment one:
A kind of method for reading data based on distributed memory system as shown in Figure 1, includes the following steps:
Step 1: receiving the read operation request that client is sent.
Step 2: counting the temperature that data are read in the read operation request.
Step 3: judging whether current operation is read operation, if so, going to step 4；If it is not, then going to step 1.
Step 4: the object data that whether there is the read operation request in pgcache is judged, if it is present going to step 6；
If it does not exist, then going to step 5.
Step 5: handling read operation request using OSD, step 4 is gone to after the completion of processing.
Step 6: the content of data is read described in assembling.
Step 7: reading and complete, return to client.
A kind of method for reading data based on distributed memory system that the present embodiment proposes, comprising: receive client hair
The read operation request sent；Count the temperature that data are read in the read operation request；Judge whether current operation is to read behaviour
Make, if it is not, then continuing to the read operation request of client transmission；If it is, judging in pgcache with the presence or absence of described
The object data of read operation request, if it is present to read the content of data described in assembling；If it does not exist, then utilizing OSD
Read operation request is handled, after the completion of processing, the content of data is read described in assembling.It can be seen that this method passes through in main pg
The upper some dsc datas of caching, and updated with the timely data aging carried out in pgcache of the reading of data and dsc data, it can
Reading delay is effectively reduced, improves the data reading speed of distributed memory system, and then improve the read-write of system entirety
Performance.
Embodiment two:
A kind of method for reading data based on distributed memory system as shown in Figure 2, includes the following steps:
Step 1: receiving the read operation request that client is sent.
Step 2: counting the temperature that data are read in the read operation request.
Step 3: judging whether current operation is read operation, if so, going to step 4；If it is not, then going to step 1.
Step 4: the object data that whether there is the read operation request in pgcache is judged, if it is present going to step 9；
If it does not exist, then going to step 5.
Step 5: the object that whether there is read operation request in pgcache is judged, if it is present going to step 8；If do not deposited
Then going to step 6.
Step 6: judging whether pgcache reaches thresholding, if it is, going to step 7；If it is not, then going to step 8.
Step 7: deleting the smallest data of temperature in pgcache.
Step 8: updating the object data of read operation request described in pgcache, go to step 4 after the completion.
Step 9: the content of data is read described in assembling.
Step 10: reading and complete, return to client.
On the basis of the above embodiments, as preferred embodiment, if when using OSD processing read operation request
It is abnormal, file of the pgcache on SSD is purged.
In addition, it is described influence data temperature operation, comprising: write, truncate, writefull, zero or
delete。
Fig. 3 is a kind of data reading system based on distributed memory system provided by the invention, as shown in Figure 3, comprising:
Request receiving module, data hot statistics module, data judgment module and data assembling module；
The request receiving module is used to receive the read operation request of client transmission；
The data hot statistics module is for counting the temperature that read data in the read operation request；
The data judgment module is used to judge to whether there is in pgcache the object data of the read operation request；
The data assembling module is used to assemble the content that read data.
On the basis of the above, further includes: object judgment module, threshold decision block, data aging module, data update mould
Block and file remove module；
The object judgment module is used to judge to whether there is in pgcache the object of read operation request；
The threshold decision block is for judging whether pgcache reaches thresholding；
The data aging module is for deleting the smallest data of temperature in pgcache；
The data update module is used to update the object data of read operation request described in pgcache；
The file removes module, if be abnormal when for handling read operation request using OSD, by pgcache in SSD
On file be purged.
A kind of data reading system based on distributed memory system proposed by the present invention can receive client transmission
Read operation request counts the temperature that data are read in the read operation request first；Then judge whether current operation is reading
Operation, if it is not, then continuing to the read operation request of client transmission；If it is, judging in pgcache with the presence or absence of institute
The object data of read operation request is stated, if it is present to read the content of data described in assembling；If it does not exist, then utilizing
OSD handles read operation request, and after the completion of processing, the content of data is read described in assembling.It can be seen that this system passes through in master
Some dsc datas are cached on pg, and carry out data aging and dsc data update in pgcache, energy in time with the reading of data
Reading delay is enough effectively reduced, improves the data reading speed of distributed memory system, and then improve the reading of system entirety
Write performance.
Fig. 4 is a kind of reading data device based on distributed memory system provided by the invention, as shown in figure 4, including
Memory and processor；
Memory, for storing computer program；
Processor is realized described in any one above-mentioned embodiment for executing the computer program based on distributed storage system
The step of method for reading data of system.
A kind of reading data device based on distributed memory system proposed by the present invention can receive client transmission
Read operation request counts the temperature that data are read in the read operation request first；Then judge whether current operation is reading
Operation, if it is not, then continuing to the read operation request of client transmission；If it is, judging in pgcache with the presence or absence of institute
The object data of read operation request is stated, if it is present to read the content of data described in assembling；If it does not exist, then utilizing
OSD handles read operation request, and after the completion of processing, the content of data is read described in assembling.It can be seen that the present apparatus passes through in master
Some dsc datas are cached on pg, and carry out data aging and dsc data update in pgcache, energy in time with the reading of data
Reading delay is enough effectively reduced, improves the data reading speed of distributed memory system, and then improve the reading of system entirety
Write performance.
With reference to the drawings and specific embodiments, the invention will be further described.It should be understood that these embodiments are merely to illustrate
The present invention rather than limit the scope of the invention.In addition, it should also be understood that, after reading the content taught by the present invention, this field
Technical staff can make various changes or modifications the present invention, and such equivalent forms equally fall within range defined herein.
Claims (8)
1. a kind of method for reading data based on distributed memory system, which comprises the steps of:
Step 1: receiving the read operation request that client is sent；
Step 2: counting the temperature that data are read in the read operation request；
Step 3: judging whether current operation is read operation, if so, going to step 4；If it is not, then going to step 1；
Step 4: the object data that whether there is the read operation request in pgcache is judged, if it is present going to step 6；
If it does not exist, then going to step 5；
Step 5: handling read operation request using OSD, step 4 is gone to after the completion of processing；
Step 6: the content of data is read described in assembling；
Step 7: reading and complete, return to client.
2. the method for reading data according to claim 1 based on distributed memory system, which is characterized in that the utilization
OSD handles read operation request, specifically includes:
Step 51: the object that whether there is read operation request in pgcache is judged, if it is present going to step 54；If no
In the presence of then going to step 52；
Step 52: judging whether pgcache reaches thresholding, if it is, going to step 53；If it is not, then going to step 54；
Step 53: deleting the smallest data of temperature in pgcache；
Step 54: updating the object data of read operation request described in pgcache.
3. the method for reading data according to claim 1 based on distributed memory system, which is characterized in that the influence
The operation of data temperature, comprising: write, truncate, writefull, zero or delete.
4. the method for reading data according to claim 1 based on distributed memory system, it is characterised in that: if in benefit
It is abnormal when handling read operation request with OSD, file of the pgcache on SSD is purged.
5. a kind of data reading system based on distributed memory system characterized by comprising request receiving module, data
Hot statistics module, data judgment module and data assembling module；
The request receiving module is used to receive the read operation request of client transmission；
The data hot statistics module is for counting the temperature that read data in the read operation request；
The data judgment module is used to judge to whether there is in pgcache the object data of the read operation request；
The data assembling module is used to assemble the content that read data.
6. the data reading system according to claim 5 based on distributed memory system, which is characterized in that further include:
Object judgment module, threshold decision block, data aging module and data update module,
The object judgment module is used to judge to whether there is in pgcache the object of read operation request；
The threshold decision block is for judging whether pgcache reaches thresholding；
The data aging module is for deleting the smallest data of temperature in pgcache；
The data update module is used to update the object data of read operation request described in pgcache.
7. the data reading system according to claim 5 based on distributed memory system, which is characterized in that further include:
File removes module, if be abnormal when for handling read operation request using OSD, by file of the pgcache on SSD
It is purged.
8. a kind of reading data device based on distributed memory system characterized by comprising
Memory, for storing computer program；
Processor, for executing the computer program to realize described in Claims 1-4 any one based on distributed storage
The step of method for reading data of system.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810929204.1A CN108958667A (en) | 2018-08-15 | 2018-08-15 | A kind of method for reading data based on distributed memory system, system and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810929204.1A CN108958667A (en) | 2018-08-15 | 2018-08-15 | A kind of method for reading data based on distributed memory system, system and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108958667A true CN108958667A (en) | 2018-12-07 |
Family
ID=64470222
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810929204.1A Pending CN108958667A (en) | 2018-08-15 | 2018-08-15 | A kind of method for reading data based on distributed memory system, system and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108958667A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109992209A (en) * | 2019-03-29 | 2019-07-09 | 新华三技术有限公司成都分公司 | Data processing method, device and distributed memory system |
| CN111240591A (en) * | 2020-01-03 | 2020-06-05 | 苏州浪潮智能科技有限公司 | Operation request processing method of storage equipment and related device |
| CN114237518A (en) * | 2022-02-22 | 2022-03-25 | 苏州浪潮智能科技有限公司 | Data reading method, system, device and terminal |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107632784A (en) * | 2017-09-14 | 2018-01-26 | 郑州云海信息技术有限公司 | The caching method of a kind of storage medium and distributed memory system, device and equipment |
| CN107656701A (en) * | 2017-09-26 | 2018-02-02 | 郑州云海信息技术有限公司 | Small documents read accelerated method, system, device and computer-readable recording medium |
-
2018
- 2018-08-15 CN CN201810929204.1A patent/CN108958667A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107632784A (en) * | 2017-09-14 | 2018-01-26 | 郑州云海信息技术有限公司 | The caching method of a kind of storage medium and distributed memory system, device and equipment |
| CN107656701A (en) * | 2017-09-26 | 2018-02-02 | 郑州云海信息技术有限公司 | Small documents read accelerated method, system, device and computer-readable recording medium |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109992209A (en) * | 2019-03-29 | 2019-07-09 | 新华三技术有限公司成都分公司 | Data processing method, device and distributed memory system |
| CN111240591A (en) * | 2020-01-03 | 2020-06-05 | 苏州浪潮智能科技有限公司 | Operation request processing method of storage equipment and related device |
| CN114237518A (en) * | 2022-02-22 | 2022-03-25 | 苏州浪潮智能科技有限公司 | Data reading method, system, device and terminal |
| CN114237518B (en) * | 2022-02-22 | 2022-05-24 | 苏州浪潮智能科技有限公司 | Data reading method, system, device and terminal |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104272275B (en) | Strengthen data buffer storage performance | |
| CN103473150B (en) | A kind of fragment rewrite method in data deduplication system | |
| US20160140054A1 (en) | Method and system for determining fifo cache size | |
| JP2014535106A (en) | Method, controller and program for populating data in a secondary cache of a storage system | |
| CN108958667A (en) | A kind of method for reading data based on distributed memory system, system and device | |
| EP3316150B1 (en) | Method and apparatus for file compaction in key-value storage system | |
| CN104063330B (en) | Data prefetching method and device | |
| CN103365954A (en) | Method and system for increasing in-line deduplication efficiency | |
| CN107209714A (en) | The control method of distributed memory system and distributed memory system | |
| CN106293525B (en) | A kind of method and system improving caching service efficiency | |
| CN104040508B (en) | For the method and system for managing data in cache systems | |
| CN106528451B (en) | The cloud storage frame and construction method prefetched for the L2 cache of small documents | |
| CN106503051A (en) | A kind of greediness based on meta data category prefetches type data recovery system and restoration methods | |
| CN103744623A (en) | Method for realizing intelligent degradation of data cached in SSD (Solid State Disk) of storage system | |
| CN111124258B (en) | Data storage method, device and equipment of full flash memory array and readable storage medium | |
| CN104050057B (en) | Historical sensed data duplicate removal fragment eliminating method and system | |
| CN103200272A (en) | Streaming media storage system and storage method | |
| US10628241B2 (en) | Determining when to send message to computing node to process items by training a machine learning module | |
| CN112051968B (en) | Kafka-based distributed data stream hierarchical cache automatic migration method | |
| CN107656701A (en) | Small documents read accelerated method, system, device and computer-readable recording medium | |
| CN107766355A (en) | Hierarchical data management method, hierarchical data management system and instantaneous communication system | |
| US20150212744A1 (en) | Method and system of eviction stage population of a flash memory cache of a multilayer cache system | |
| US8380962B2 (en) | Systems and methods for efficient sequential logging on caching-enabled storage devices | |
| US10747596B2 (en) | Determining when to send message to a computing node to process items using a machine learning module | |
| Kaczmarczyk et al. | Reducing fragmentation impact with forward knowledge in backup systems with deduplication |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20181207 |